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ABSTRACT 

A 2-week institute trained 30 public school 
personnel, all involved with the Illinois gifted program, as 
evaluators. An evaluation design, materials, and instruments were 
developed; the workshop evolved from evaluation planning to skill 
training in data gathering techniques. Evaluation in terms of each 
participants local situation was stressed. Concerning the 
participants* progress, results' indicated significant shifts in 
attitude toward evaluation as well as significant gains on the 
achievement test. The fact that the completed evaluation plans could 
be used by the participants was considered a meaningful success. 
Problems involved interaction, heterogeneity, and continued 
direction; suggestions were made for improvement. Over half of the 
document consists of materials and instruments used. A detailed text 
describing the training materials of evaluation used at the institute 
is available as EC 005 463. (JD) 



1 



r- 

uPv 

o 

cr 

r<"\ 



ILL IcJPURi' Ji-; T^L 1968 SUMMER INSTITUTE ON EVALUATION 



UNIVERSITY OF ILLINOIS 




July 29 - August 9, 1968 



Ui 



SPONSORED BY: 

COOPERATIVE EDUCATIONAL RESEARCH LABORATORY, INC. (CERLI) 

Northfield, Illinois 

CENTER FOR INSTRUCTIONAL RESEARCH AND CURRICULUM EVALUATION (CIRCE) 

University of Illinois, Urbana 



SUPPORTED BY: 

DEPARTMENT OF PROGRAM DEVELOPMENT FOR GIFTED YOUTH 
OFFICE OF THE SUPERINTENDENT OF PUBLIC INSTRUCTION 

STATE OF ILLINOIS 




VJ 





THIS DOCUMENT HAS BUN REPRODUCED EXACTLY AS RECEIVED FROM IHE 
PERSON OR ORGANIZATION ORIGINATING II POINTS Of VIEW OR OPINIONS 
STATED DO NOE NECESSARILY REPRESENI OFFICIAL OFFICE Of EDUCATION 
POSITION OR POEICV 



OCTOBER, 1968 

U S DEPARTMENI Of HEALTH. EDUCATION & WELFARE 
OEEICE Of EDUCATION 



EVALUATION BY: 



CERLI 



EDO 39657 



INTRODUCTION 



l 



» 

1 




l 





The Cooperative Educational Research Laboratory, Inc. (CERLI) and the Center 
for Instructional Research and Curriculum Evaluation (CIRCE) cosponsored 
the Summer Institute on Evaluation conducted at the University of Illinois 
(Urbana) July 29 - August 9 and supported by the Department of Program 
Development for Gifted Youth, OFFICE of the SUPERINTENDENT of PUBLIC INSTRUC- 
TION, STATE of ILLINOIS. 

At the multi-purposed institute, the cosponsoring agencies demonstrated the 
feasibility of training nonresearch personnel to conduct evaluations, pilot 
tested a two-week training package and provided inservice training for the 
Gifted Evaluation staff. 

In this document, the CERLI staff reports the planning, presentation and 
evaluation of the Summer Institute on Evaluation. 
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( Parts I & II ) 



I. PLANNING THE SUMMER INSTITUTE ON EVALUATION 



The institute was designed and presented to achieve the following purposes ? 



To provide for evaluation of local gifted programs 
within the Illinois Plan 



To build a continuous assessment capacity into the 
Illinois Plan by training a cadre of evaluation leaders 



To demonstrate the feasibility of training nonresearch personnel 
to conduct evaluations 



To produce and field test an extensive set cf evaluation training 
materials that can be used around the state and around the country 



To provide inservice training for the Gifted Evaluation staff. 



Objectives for the participants included: 

Stimulus objectives : 

It was intended that each participant would encounter: 

The logical arguments for gathering evaluation information 
to assist in the making of rational decisions in the 
local project 

An evaluation model around which the local evaluation 
efforts could be organized 

Discussion of evaluation problems featuring the viewpoints 
of experienced evaluators, researchers, and administrators 

A growing opportunity for sharing local project findings 
with staff members of other projects 

Direction to an abundance of resources, personal and otherwise, 
to aid in the conduct of evaluation at the local level. 
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Performance objectives: 



It was expected that each participant would: 

Design components of an evaluation plan for local use 

Make a table of contents for a summative evaluation report 
of his own local gifted program 

Improve his facility for using the language and concepts 
of measurement and evaluation 

Prepare to try ou‘: certain standardized classroom observation 
techniques that can be useful in evaluation studies 

Examine prototype evaluation reports and read selections 
from the literature on evaluation 

Work out solutions to a series of problems designed to 
simulate the conditions and circumstances of local 
gifted projects in Illinois. 

Rationale for Design 

Stake's "The Countenance of Educational Evaluation" provided the institute's 
conceptual framework for this model projects a comprehensive and interrelated 
view of evaluation that seemed desirable for the participants to acquire. 

It also seemed important to involve participants in experiences related 
to the actual problems confronting them in their local school districts. 

Thus, the design for the institute synchronized theory and practice in 
order that the trainees might develop skills and collect materials they 
could utilize and apply in solving local problems. 

During the first week, each participant was expected to develop an 
evaluation plan. In the initial sessions. Stake presented his model to 
orient the participants to evaluation and suggest types of data they might 
use. Model plans, embodying these ideas, were distributed. The remaining 
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sessions dealt with various topics (classroom observation, research design, 
statistics) which would help participants to write their plans. 

During the second week, it was hoped that participants would develop 
research skills that they would need to use in implementing their evaluation 
plans. In this context, continued work in statistics, test development, 
scale construction, questionnaire construction and interview training 

were presented or available. 

♦» 

Participants 

Because the institute was so closely associated with the Gifted Child 
Program, individuals directly involved in it initially were contacted: 
Demonstration Center Directors, Reimbursement Center Directors, Experimental 
Project Directors and state staff in charge of these projects. Upon 
request, the Director of the State Title IV project recommended some local 
Title III Directors whose backgrounds included research methods. These, 
individuals seemed particularly eligible for the institute since they were 
involved with innovative programs that had been only superficially evaluated, 
if at all. CEELI sent a letter and brochure (Appendix 1 and 2) to each of 
these potential participants. 

In all, some 500 letters and brochures were mailed and some 100 recipients 
responded by requesting application blanks (Appendix-3). Forty applications 
were completed and returned and from this number 33 applicants were selected. 

Three criteria determined the acceptability of applicants: background in 

academic courses; future job responsibility for conducting an evaluation 
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in a local district; experience in research and evaluation projects Among 
the applicants, some teachers were accepted even though there had been some 
effort to restrict enrollment to administrators. The fact that the teachers 
might have some job responsibility for evaluating thou..: programs during 
the coming year and the fact that there were so few applicants altered the 
original selection scheme. 



After analyzing the selection process, the CERLI staff offers the following 
suggestions for improvement: 

1. In order to more precisely delineate an applicant’s background, the 

application form should include: categories identifying major fields 

of study and graduate hours in statistics, tests and measurements, etc.; 
specific information about past and current professional experience and 
positions. 

2. In order to determine an applicant's potential, some type of 

d iagnos tic test might be constructed and administered to all applicants. 
With this criterion, background and position then would become secondary 
items in the selection process. 

3. Since teachers ordinarily do not have the authority or the resources 
available to evaluate a program, some prior commitment should be made 
by their administrator who could authorize their active participation 
in an evaluation. Otherwise, teachers attending the institute could 

be frustrated if they cannot use the skills they have developed and their 
participation may actually have been or seemed a wasteful effort. 



II. PRESENTATION of the SUMMER INSTITUTE on EVALUATION 
Basic Plan 

To inform participants about the workshop program and its continuity and 
content, a preliminary schedule was prepared and sent to them. 

During the first week, all participants (either individually or in small 

groups) worked on developing a plan to collect information on some specific 

question. Presentations dealt with concepts or ideas related to the problem 
of developing a plan. 

During the second week, participants worked with individual problems such 
as statistical exercises and development of instruments for their' evaluation 
plans. Formal presentations dealt with the development of skills that an 
evaluator uses to solve problems of data collection. 

Daily Activities and Historian’s Commentary 

Each of the following "daily reports" presents the statement of objectives 
that participants were given and the historian's account of the activity, 

(See pages 13 -14 for a graphic comparison of the planned and the actual 
schedule that evolved.) 

WEEK ONE 
Monday 

The morning sessions will be for introductions and an orientation to the 
v7or s op. In these sessions we hope to elicit from you the general 

and specific evaluation problems that concern you and the expectations 
you have of the workshop. 
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Bob Stake will make the first of two presentations of his evaluation model 

in the first afternoon session. These presentations (the second is on Tuesday a.m.) 

will provide you with a general overview of the model. Subsequent presentations 

will be relevant to specific components of the model, and they will assist 

you in translating the model into a specific plan. You should have read the 

article "The Countenance of Educational Evaluation" before hearing the 

presentations by Dr. Stake. 

Three videotapes of consultation sessions between a school person and an 
evaluation expert have been prepared. Each tape projects a session that was 
held prior to the development of one of the three evaluation-plan examples. 

Showing of one of these tapes has been scheduled for the fourth session on 
Monday . 



Changes: The evaluation model was presented by Dr. Stake at the second session 

at 10:15 and the expectations of the participants were elicited in the small 
groups during the afternoon. This was done because a faculty meeting scheduled 
for the afternoon required Dr. Stake's presence. There is little evidence 
for judging whether or not this change made a difference except that the 
participants stated that they wished there had been a chance to meet informally 
that first morning. The expectations session might have provided this oppor- 
tunity. Also It is unclear whether or not Stake's presentation of his model 
might have confused some of the participants to the extent that their expecta- 
tions became fuzzy. 



Tuesday 

A continuation of the presentation on the Stake model will come in the first 
Tuesday session. 

The second session will be a work session during which you can begin work 
on your evaluation plan. You may want to study the example plans at this 
time. 
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Material of relevance to the observations cells of the model will be pre- 
sented in the third session. Topics that will be considered in this session 
include operational definitions and principles of test selection. 

The first part of the fourth session on Tuesday will be a presentation on 
the use of certain resource materials such as The Mental Measurements 
Yearbook , Research in Education , and Review of Educational Research . The 
remaining time in the session will be a work session. 

Tuesday's fifth session is planned as the time when Dr. Stake will use 
one of the videotapes to discuss the role and task of the consultant. 



Changes: The second session was changed from an individual work session 

to a group work session in which the CERL1 staff attempted to act as 
catalysts in having the participants discuss Stake's model and react to 
it according to their own planning problems in the hope that they would 
develop a better understanding of Stake's model through group discussion. 

Dr. Denny brought the participants to the ERIC research center on the 
U. of I. campus for his fourth session discussion on resource material. 

The operation of the center was explained by the staff and a tour of the 
facility was arranged. Explanatory materials were distributed along with 
a promise of cooperation if any of the participants wished to use ERIC's 
facilities in the future. The reaction seemed mixed according. to background. 
Those participants with a background in research methods were much more 
impressed than those who had little training in this area. 



Wednesday 

A second presentation on conditions of observation is scheduled for the 
first Wednesday session. Dr. Denny will discuss classroom observation 
procedures. 



The second and third sessions are scheduled as work sessions. Hopefully 
by the end of the third session you will have defined a rough outline 
of your evaluation plan. The rest of the videotapes will be shown on 
a schedule from 10:15 - 3:00. 

The format of evaluation reports will be presented in the fourth session. 



Changes: The fourth session was changed from "Report Format" to "Textbook 

Evaluation." Dr. Stake thought that the participants were not ready for 
his lecture on report format at that time. Instead, a graduate student 
spoke about textbook evaluation. The change definitely brought about 
positive and negative reactions. The positive reaction was that the par- 
ticipants were able to come out of the protective shell which they had 
constructed to avoid interaction with the professors. The negative 
reaction was that the participants reacted very strongly against the un- 
substantiated opinions that the student had made, and they thought he 
had nothing at all to contribute to them. 



Thursday 

The first session will be devoted to a presentation on ways to establish 
standards as bases for judgments. The content will be primarily on re- 
search designs that might be used in evaluation situations. The validity 
of the data obtained in the various designs will be stressed. 

The second session will be a presentation on observational procedures 
not commonly used but with rich potential. The topic is unobtrusive 
measures . 

The third and fourth sessions are planned as work sessions. Hopefully 
your plans will be ready to be submitted to the staff at the end of 
session four on Thursday. 

The staff will develop some artificial data for your plan on Thursday night. 



These data will be used by you in the Friday exercise. Your plans will also 
be read by the staff and feedback provided you by Monday. 

(Note: the exercise with artificial data was not assigned as the director 

thought the work might be "too advanced" for participants to handle.) 

The fifth session will be a presentation on statistical problems. This 
presentation and the first one on Friday will provide an overview of 
properties of scales and of certain statistical techniques. 



Changes: Dr. Stake asked his graduate students who are preparing for their 

Ph.D. to act as consultants to those participants who wanted more individual 
help in writing their evaluation plans. The sessions seemed profitable 
for the participants and for the students who received practical consulting 
experience. 



Friday 



The first session is planned as a presentation on statistics. 

The second session is a work session in which you will analyze the data 
provided you and prepare a report of your evaluation. Much of the report 
have been written as the material prepared during the week. The 
reports will be duplicated over the weekend and distributed on Monday. 

The third session is scheduled as a time for evaluation of the first 
week s activities in the workshop. You will be asked to complete some 
evaluation instruments } but we also hope you will discuss the strengths 
and weaknesses of the week as you perceived them. The planned schedule 
for the second week will be presented and perhaps revised on the basis 
of your expressed interests and desires. 



Changes: During the second session one of the statistical exercises was 

distributed for individual and group practice. Since the vast majority 
of the participants had completed and handed in their plans by this time, 
the second session activities were eliminated and the third session 
activity of interviewing the participants was moved up. Therefore, the 
first week's session was ended by noon. 
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WEEK TWO 



Monde./ 

The first session on Monday is an orientation to the second week. This will 
be needed especially if changes are suggested by points you raise in the 
Friday evaluation session. If time permits, part of the first session will 
be available for you to think about your individual activity for the second 
week. Feedback on your plans may also be provided. 

The second session is planned as a presentation on judgments. Techniques 
for making judgments will be covered. 

The third session is planned as a panel to bring up and discuss problems 
that confront an evaluator such as inadequate questionnaire returns, 
administrator interference, etc. 



The orientation for the second week's activities lasted only thirty minutes. 
The remainder of the time allotted for the first session was spent with role 
playing in small groups. One participant would explain his evaluation plan 
(which had been handed back with comments but not duplicated because of a 
lack of facilities) and the others would ask questions while playing the 
roles of parents, administrators, teachers, etc. 



Tuesday 

The Tuesday presentations are on test construction. Topics to be covered 
are measuring achievement, measuring higher-order mental process, measuring 
attitudes and scaling. Principles of test construction will be stressed. 
The presentations will be in the first, third, and fifth sessions. The 
second and fourth sessions will be work sessions. 



Changes: Dr. Stake recommended that Mrs. Susan Feldman lecture on question- 

naire construction during the third session because of the relevance of the 



topic to her present work at the University. 



The evening session on scaling was moved to the second session on Thursday 
morning for two reasons: first, many of the participants had stated weariness 

from night session during the first week; secondly, the voting at the Re- 
publican convention occurred this night and possibly the attendance and the 
interest of the participants could have been affected. 



Wednesday 

The first session on Wednesday is planned as an overview of survey procedures. 
The presentation will include material on kinds of information usually 
obtained in surveys and sampling considerations. 

The construction of questionnaires and interview schedules is the topic 
assigned to the second Wednesday session. 

The Wednesday afternoon and evening sessions are scheduled for an interview 
training exercise. Dr. Denny will conduct these sessions and will structure 
the situation for you. 



Changes: Dr. Denny recommended that Dr. Gene Summers , a sociologist at the 

University's Survey Research Laboratory, lecture on "Survey Procedures" 
because this area is his speciality. 

Dr. Stake gave his lecture on test development which had been replaced by 
Susan Feldman's talk the previous day. 



Thursday 

The first two sessions on Thursday are planned as work sessions. 

The Thursday afternoon sessions are planned for the second part of the 
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interview training activities. The content of these sessions will follow 
up the Wednesday afternoon presentation. 



During the afternoon sessions, the CERL1 staff conducted the remaining 
interviews with the participants. This was arranged to avoid the possibility 
of missing this information in the rush of data collection Friday. It became 
feasible since Dr. Denny was able to work with only a few people at a time 
in his interview training. 



Friday 

In the first session on Friday we have planned a critique and discussion 
of the evaluation- pi an handout on the question "Has the gifted program 
had an effect on the achievement of the participating students?" 

Dr. House will present some ideas on the establishment of an information 
pool in the second session. The information pool would be a central 
storage and clearinghouse of information on the gifted program in the 
state. 

The third Friday session is planned as an evaluation session and one 
in which the end-of-workshop administrative details are handled. The 
workshop will close at the end of this session. 



The institute ended at noon. All the necessary data was collected after 
George Trout's speech. 
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HASTINGS 



TRAINING MATERIALS 



Pre-Package 

Description of Program and Description of Workshop Exercises 
Robert Stake "The Countenance of Educational Evaluation" 

List of References 
List of Participants 

Week #1 Package 

Index 

Revised list of participants 
Viewing the Videotapes 
Questions for videotape 

Stufflebeam "Developing Evaluation Design" 

Evaluation Plan I: "Does the gifted program in our school increase the 

student s ability to conduct independent study?" 

Evaluation Plan II: "Which of these three sets of science materials should 

we use in our elementary science course?" 

Evaluation Plan III: "Are we selecting the right students for our class in 
creative writing?" 

Suggestions for Role Playing 
Week //2 Package 

Garth Sorenson "A New Role in Education: The Evaluator" 

Exercise on Rating Scales 

Attitude Scale Exercise 

Glossary of Statistical Tferms 

Computation of Chi-Square 

Computation of Pearson r 

Computation of Spearman rho 

Computation of t— test and Mann— Whitney U 

Computation of Correlated t-test 

One way and two way analysis of variance 

Evaluation Plan IV: "Has the gifted program had an effect on the achievement 

of the participating students?" 

Accessories 



Video Tap e I — Bob Stake and graduate student 

"Does the gifted program in our school increase the student* s ability 
to conduct independent study?" 

Video Tape II - Terry Denny and graduate student 

"Which of these three sets of science materials should we use in our 
elementary science course?" 

Video Tape III - Tom Hastings and Douglas Sjogren 

"Are we selecting the right students for our class in creative writing?" 
Video Tape IV - Bob Stake and School Personnel 
"The Evaluator Role" 
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Booklet 

Cook, Desmond. Program Evaluation and Review Techniques: Applications 

in Education. Washington: U.S. Government Printing Office, 1966. 



Periodicals 

"improving the 'Natural System 1 : EPIES Three Year Plan," The EPIE Forum , 

Vol. I, #5, January, 1968 

O'Brien and Lopate. "Preschool Programs and the Intellectual Development 
of Disadvantage Children." Occasional Paper: ERIC . Urbana, Illinois: 

University of Illinois Press, 1968 

Pamphlets 

Griffin, Louise. "How to Use ERIC", 1968 Hjelm, Howard; Sherman, Marian. 
"The National Laboratory on Early Childhood Education: A Model of 

Educational Research and Development". Athens, Georgia: College of 

Education Publishing 

University of Illinois, "Survey Research Laboratory" U.S. Department 
of Health, Education, and Welfare. "ERIC Can Help" 

Speeches 

Howe, II, Harold. "Picking Up the Options;" 

Monographs 

Goheen and Davidoff. "Biserial Correlation Graph for Test of Significance" 
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III. EVALUATION of the SUMMER INSTITUTE on EVALUATION 



Scheme 

For our evaluation design, we are indebted to the evaluation of the 1966 

t 

AERA Presession. Although we did not intend to emulate this design, its 
logic became more compelling as we thought about what we wanted to know and 
as we developed our design. For us, however, concern about the long term 
development of our materials dictated the collection of detailed information 
about their use. In this respect, our schemes differ. 

To determine the immediate overall effect, we developed an achievement test 
and an attitude inventory on the evaluator role and administered them on a 
pre and post basis. Though we had no true comparison group, we used an 
advanced university class as a rough comparative measure on achievement. 

The Participant Opinionnaire gave us the participant’s overall reaction 
to the institute. The Participant Critique Form (which we copied almost 
intact from the Presession Evaluation) gathered participant reaction on 
more specific features of the institute e.g. the structure, interaction, 
and physical setting 0 These two instruments were post administered. 

In order to ascertain the worth and impact of specific materials, video 
tapes, certain lectures, etc .-a critical input for further developmental 
work- we devised a participant interview schedule that involved interviewing 
half the participants during the first week and the others, the second 
week. Although this instrument somewhat duplicated items in the Opinionnaire 

"'Report of the AERA 1966 Presession on Experimental Design 
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and the Critique Form, such redundancy was not objectionable as the data 
were soft, the instruments untested, and the participants' reactions subject 
to fluctuation over time.. 

Realizing that instruments cannot and do not record many significant events 
that occur at such institutes, we used an unidentified observer-historian 
to collect data. These observations, corroborated by other instruments, 
proved to be very valuable — particularly in dealing with the lack of social 
interaction, the least effective phase of the institute. By recording and 
succintly describing what we did, the observer also made a valuable contribu- 
tion since, quite naturally, our actions differed from our intents. 

Possibly the most important immediate indicators of the institute's success 
are the evaluation plans that the participants produced. (See pages 45-60 ) 

Although our immediate evaluations of these plans could only be rough 
estimates of success, through follow-up sessions with the participants we 
will be able to evaluate the long range effects. Ultimately, an excellent 
evaluation of a local program will provide a criterion of success; but we 
shall not impose high standards for no two-week institute could resolve 
the deeply rooted problems in evaluation that for too long has been 
virtually neglected. 

The evaluation of immediate effects documents the success of the initial 
step, the training of evaluators. In order to evaluate long term effects, 
we must acknowledge and cope with the problems that conditions within local 
districts generate. 
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The Observer’s Report 



Introductory note; 

One main participant observer attended all sessions and completed observa- 
tion schedules for each. In an attempt to produce a reliable score, 
his ratings were compared with two other CERLI staff members who alternately 
rated sessions * 

Ratings of the tapes and lectures reflect the observer's judgments and 
informally gathered participants' reactions and attitudes. 

Information about the small group work sessions was collected and studied 
by the three CERLI staff members who informally interacted with participants. 
What the participants communicated (either on the critique or during 
the structured interview) seemed to validate the main participant 
observer's report and the data other instruments provided. 

The observers' instruments (roughly based on the observer's report of 
the AERA 1966 Presession) were extensively revised to complement the 
style of this institute. 

See Appendix-4 for schedules and proposed revisions. 



Institute Organization 

The best description of the institute's atmosphere probably would be that 
of flexible but structured. Daily changes in the schedules (pages 13 & 14 ) 
reflect its flexibility. However, the observation schedules show that the 
starting times were strictly adhered to and that the participants' atten- 
dance was extremely high and punctual. 



The Participants 

Attempting to find a common feeling or attitude toward various aspects of 
the institute (such as the level of difficulty of a particular lecture) 
often was impossible because of the participants' diverse backgrounds. 

When the observer did elicit their common reactions, only such matters 
as facilities would be verbalized. Perhaps the participants' reactions to 
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the lecture on statistics best illustrates the observer’s difficult task: 

About one third of the participants (whose statistical background was 
minimal, if any) considered the lecture too difficult to comprehend. 
Another third (whose background was extensive) considered the lecture 
time-wasting because the lecturer dealt only with theoretical aspects 
and did not explain how to use this knowledge in working with their 
evaluation plans. The other participants (who had tafcen some basic 
courses) appreciated the opportunity to review and update their 
fundamental knowledge. 

Although all participants admitted that each speaker (with one exception) 
had taught them something, their diverse and individual backgrounds 

rather than the presentations or material apparently modulated their degrees 
of satisfaction. 

Facilities 

If listed on a continuum, comments about facilities would be predominately 
negative. The fact that there were no desks and that the chairs provided 
no writing surface complicated the extensive note-taking that participants 
needed to do. The room was small and, for some of the men, the space 
inadequate for comfortable "leg room". Speakers and participants criticized 
the inadequate blackboard space which was confined to a single, portable 
model. 



During the second week, participants seated in the back rows were 
continuously distracted by noise from an adjacent room. On three occasions, 
staff members tried to alleviate the problem. Only by reserving the adjoin- 
ing room for the same two-week period and using it for small group sessions 
when needed could the problem have been solved. 

On one occasion, TV monitors set up to show various video tapes obscured 
the lecturer’s notes on the blackboard. Otherwise the placement and use 
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of monitors were effective. 



Although participant interaction was adequate, it could have been much better 
and it could have occurred sooner. This observation was validated by casual 
comments on the last day when several people admitted they still didn’t 
know the names of all the participants which then numbered 25. (Five 
applicants did not attend; two left during the second week to fulfill 
job commitments; one left because of illness.) 

Participants thought they should have worn name tags the first morning and 
that an informal coffee and roll break would have prevented them from cluster- 
ing in small groups of friends or strolling alone during the break. They 
also thought that some time for the group to informally meet each other and 
discuss the orientation session, their individual hopes and fears about the 
institute, etc. should have been planned. These matters not only were 
discussed informally with the observers later in the institute, but also 
formally expressed in some of the structured small group sessions. 

Ideally, strong relationships could have been established in the small 
groups. Originally, each group included some members whose ability and 
potential leadership were outstanding and a member of the CERLI staff who 
would function as a catalyst or a recorder. The other participants were 
assigned to one of these five groups. 

Actually, the groups had little opportunity to form a strong and cohesive 
unit. In the second meeting on Tuesday morning, groups I and II and 

groups III and IV merged because there were not enough available rooms to 
accommodate each group. 
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At the next opportunity for small group meetings, the speaker asked 
participants to form small groups among those seated near them to discuss 
possible ways of taking unobtrusive measures. Such shifting of personnel 
may have been a factor in the lack of leadership development among the 
participants. Whatever the reason, CERL1 staff members had to assume more 
and more control in helping the groups focus on and accomplish their 
respective tasks. 

Another factor affecting group development was the fact that the structured 
purposes of the meetings often changed or at times seemed very general. 

This flexibility contributed to some groups 1 ineffectiveness because the 
CERLX staff neither had anticipated nor had been trained for this role 
in group leadership. Some participants thought that if speakers had 
worked with various small groups, problems in speaker-participant rapport 
and group leadership might have been alleviated. 

Lecturer-Participants Interaction 

Because interaction between lecturer and the group often did not occur, 
many of the structured responses on the observation schedules were marked 
"inconclusive". For example, if participants asked only one or two 
questions, the relevance or depth of the discussion motivated by a lecture 
could not be conclusively stated. 

(In some instances, the audience did not respond because the speaker posed 
either rhetorical questions or questions eliciting only a negative response.) 
However, speakers who involved the group generated positive reactions and 
motivated questions. For example, interaction occurred during Dr. Stake's 
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lectures on test construction and Q-sorting (scaling) and Dr. Denny’s talk 
on unobtrusive measures. 

During tne early sessions of the first week, the participants’ insecurity 
among their peers and their lack of experience with evaluation as presented 
at sucli a tiieoretical level prevented many of them from asking questions. 

One lecture that did activate the group was a graduate student's presenta- 
tion of textbook evaluation. Although the group generally agreed that 
tne lecture had not been particularly informative, the opportunity to 
challenge a speaker who seemed unsure of himself and who (in their opinion) 
presented misinformation encouraged participants to express their ideas to 
the group. After this experience or critical incident, participants remained 
unresponsive only once. For the most part, only lack of time terminated 
interaction or discussion during the rest of the institute. 

Participants' Response 
Achievement 

The staff prepared but had no opportunity to try out the achievement test 
before the institute. In constructing the 20-minute test, the staff 
attempted to relate lecture content to the 46 test items. 

Administered both as a pre and post measure, the instrument provided 
the following data. 

After the 2-week session, the 3.40 gain in test scores was significant 
at the .01 level. However, the pre-test score might reflect the fact 
that participants had had an opportunity to read Stake’s "The Countenance 
of Evaluation" before coming to the institute and to the extent that 
they had read and absorbed this material (included in test content) 
their pre-test scores would be influenced. 
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To compare the post-test performance of the group. Dr. Stake asked students 
who recently had taken his course in Educational Psychology at the University 
of Illinois to take the test. However, the pseudo-control group proved to 
be very small as only five students responded to the mailed request and 
enclosed test. The 5 tests indicated a mean slightly higher than the 
institute participants but beyond that any further interpretation would 
be meaningless. 

Using the split-half method of determining reliability and the Speakman- 
Brown prophecy formula, analysis of pre-test results shows reliability 
to be .71. An analysis of variance performed on the same data indicated 

reliability to be .68. 

However, item analysis based on the 7 upper and 7 lower scores in the 
group indicated 14 of the 46 items within acceptable difficulty and 
discrimination levels. 

Five of the items (representing 50% of this sub-test area) appeared 
in the Multiple Choice section of the test. 

The minimal background material on participants prevented an analysis of 

effects of background variables on performance and thus weakened score 

interpretation. More background material also would have been useful in 

examining attitude inventory. 

One participant commented that the post-test created some anxiety during 
the last two days of the institute. Others felt that the pre-test some- 
what structured the material to be learned during the two weeks. 

(For a copy of the test and additional information, see Appendix __ .) 
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Attitude 



The attitude inventory (Appendix u) was built on the basic assumption that 
educators' negative or positive reactions to given statements of opinion 
would provide their perceptions about evaluation. In this instrument, 
statements were meant to convey the role of an evaluator and the nature 
of evaluation. For the most, part, the statements were culled from positions 
made by Stake, Scriven or Sorensen; a few were provided by the authors of 
the instrument. 

In order to discover what change in attitude occurred during the two weeks 
of the institute, the attitude inventory was designed as a pre and post 
instrument. (According to the evaluation of the 1966 AERA presession, 
changes in any category reflecting differences of 15% between pre and post 
measures could be considered significant.) 

In this instance, however, measuring change was complicated by the fact that 
the attitude inventory included many items reflected in Stake's "The 
Countenance of Evaluation" that participants had received and presumably 
read before coming to the institute. In spite of this potential influence, 
however, the following fourteen items did show a significant shift from 
pre to post in the various categories: SA (strongly agree), A (agree), 

2. (undecided), D (disagree), and SD (strongly disagree). 
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(Pre-test percents appear above the responses; post-test, below ) 

1. "The role of the evaluator should be that of a describer rather than 
a grader." 

• 

31 50 8 8 3 

SA A ? D SD 

46 42 0 12 0 

After the institute, more participants strongly agreed with this 
statement, but the shift was not a dramatic one. 

5. "One of the first things an evaluator must do is obtain a list of 
behavoral objectives." 

12 42 23 19 4 

SA A ? D SD 

19 50 8 19 4 

The undecided category showed a shift in this item. There were fewer 
individual participants in this category after the institute than before. 
The shift was to the slightly agree and strongly agree categories. 

7. "Evaluators often pay too much attention to what they have been urged 
to look at, and too little attention to other facets." 

35 42 12 11 0 

SA A ? D SD 

23 58 12 7 0 

More people agreed with this statement after the institute, although 
the shift came basically from the strongly agree category. 

8. "The kind of data gathered in an evaluation should seldom be determined 
by what the groups are like that will receive the results of the 
evaluation." 

12 8 19 35 26 

SA A ? D SD 

4 12 0 19 65 

There were significant shifts in categories on this item. At the end 
of the institute, no one was undivided on the question and the big shift 
moved toward the strongly disagree category. 
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"As long as hoped for outcomes occur, it is not important that 
objectives be stated clearly." 



0 

SA 

8 



4 

A 

15 



0 

9 

9 

4 



54 

D 

35 



42 

SD 

38 



The shift took place in the slightly disagree category. After the 
institute,, many participants shifted from this category to a more 
positive position. 

10. "The most important use of evaluation findings is to change the program. 
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More were in slight agreement with the item after the institute. 

12. "It is possible to evaluate a program without knowing the goals of 
the individual teachers." 
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SA 

8 



12 

A 

31 



19 

? 

9 
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31 

D 

38 



34 

SD 

19 



Significant differences occurred in three categories after the institute 
Few were undivided after the institute; more were in slight agreement 
with the statement; and fewer strongly disagreed. 

13. "The personal characteristics of the evaluator are a major determinant 
of the evaluation." 



0 

SA 

8 



27 

A 
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19 

9 
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38 

D 
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16 

SD 
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A shift occurred within two categories on this statement. More slightly 
agreed and fewer slightly disagreed. 
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"It is not practical to draw conclusions in evaluating a program 
prior to the program's completion." 
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A shift took place between the slight agreement and slight disagreement 
categories. Few slightly agreed after the institute; more slightly 
disagreed. 

17. "It is up to the local educator to rule out the study of a variable 
because it is not one of his objectives." 
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The shift in this statement took place between the undecided and slight 
agreement categories. After the institute no one was undecided; more 
slightly agreed* 

18. "No school can evaluate the impact of its program without knowledge of 
what other schools are doing." 
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Here the shift took place between extreme categories. From slight 
agreement, the shift was toward the disagreement scale, with a number 
of participants strongly disagreeing; no one had strongly disagreed 
on the pre-attitude inventory. 

21. "An evaluator has the right to decide what to evaluate." 

4 34 15 35 12 

SA A ? D SD 

4 54 4 35 3 

On this statement, more slightly agreed with the item after the institute. 

25. "Absolute standards, e.g. the judgments of people, should not be applied 
to a program." 

4 8 8 69 11 

SA A ? D SD 

0 4 4 62 30 

More participants strongly disagreed with the statement on the post- 
attitude inventory. 
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In addition to comparing the changes that took place in the participants 1 
attitudes before and after the institute, another comparison between the 
attitudes held by an expert and those held by the participants can be made. 
Ideally, one would hope that there would be a shift in the participants 1 
attitudes after the two week session toward those held by the expert. 

This in fact did occur. 

Though the expert’s ratings or attitudes do not reflect the entire staff’s 
thinking, they do provide some absolute basis for comparing the intended 
and actual outcomes of the program. In this case, the expert was a member 
of the institute’s evaluation staff. 

For comparative purposes and for the purpose of performing a statistical 
test of significance, the pairs of categories in a like direction 
(SA & A, D & SD) were merged. For example, a rating of A (slightly agree) 
given by the expert encompasses the rating SA (strongly agree) or vice 
versa, and a rating of D (slightly disagree) also encompasses SD 
(strongly disagree). The category of ? (undecided) is treated as a 
single unit. Thus, if the expert marked a statement with A, the 
participants* reaction in both A and SA categories is merged in one 
category for comparative purposes against the rating given by the expert. 

For example, the expert indicated slight agreement with statement //I 
(shown in the Appendix 6) and 81% of the participants either slightly agreed 
or strongly agreed on the pre-attitude inventory (31% of the participants 
marked SA and 50% marked A). After the institute, 88% either slightly 
agreed or strongly agreed (46% *for SA and 42% for A). Thus, 81% on the 
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pre-test and 88% on the post-test then are considered as agreeing with the 
expert— a 7% gain after the training session. 

During the session, participants 1 evaluation attitudes on the attitude 
inventory, taken as a whole, shifted toward those held by the expert. 

This pre to post attitude inventory shift tov/ard the area that the expert 
considered desirable was significant at the .05 level. This change was 
equivalent to the shift of almost two individuals moving across all items 
toward the expert. 
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Opinionnaire 



Participants' response to the opinionnaire (administered at the close of 
the session) provided information about the institute's physical facilities, 
content, relevance. Most participants considered the housing and dining 
facilities very satisfactory. The meeting and work rooms, however, were 
less satisfactory. 

On the whole, the participants seemed quite satisfied with the institute's 
content. They did suggest, however, that activities could be better 
organized; that scheduling could be improved; that small groups be 
assigned specific tasks; that more practice in developing instruments would 
be beneficial. One participant felt that alternative approaches to evalua- 
tion also should be presented at the institute. 

V 

For the most part, participants felt thai: the institute related to their 
interests and background and that the experience had been valuable. They 
wished to pursue (either independently or at another workshop) topics 
that had been introduced at the institute. Of particular interest were 
statistics and Stake's evaluation model. 

Specific responses to the opinionnaire : 

1. Did you have enough information about this workshop before 

you arrived? Yes: 17. No: 4. 

2. (If no) What else would you like to have known about? 

All of the references. Each objective at the beginning 

of the institute. 

3. There are many parts of a Workshop experience that can either 
contribute to your satisfaction or detract from it. For each 
of the following, would you let us know how satisfied you've 
been? 
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1. Really outstanding 

2, Very satisfactory 

3. Just acceptable 

4, Reed improvement 

NR/MA. Wo response/not applicable 



a. Weals 



b. Hotel rooms 



NR/NA 



« 4 a f 



• 9 • 



• f O 



c. Meeting rooms 

d. Other facilities or services 



e. Facilities for working . , . 

f. Opportunity for discussion . 

g. Presentation in general , . . 



16 



14 



10 



16 



13 



8 



16 



(If you have checked "need improvement" for any of the foregoing, 
please note below any suggestions you may have.) 

Library needs to be more centrally located. 

More time — help on problems. 

More group mixing needed. 

Larger meeting room with tables. 

More person to person contact. 

Grouping on basis of problem areas. 

Snail group feedback after each major presentation. 

4. Would you describe the one or two most valuable ideas that you 
received from attending the Workshop? 

.. Survey research methodology. 

.. Evaluation is not an exact science. 

.. Stake model for sifting data. 

.. Evaluation as a describes 

.. Information on scaling, questionnaires, etc. 

. . Interview training. 

o. Subjective data can be classified as to be useful for evaluation, 
. « Evaluation is continuous. 

. . A small part of the program can be evaluated. 

5. As far as you're concerned, what would have most improved the 
Workshop? 
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.. More interaction among participants. 

. . Alternative approaches to evaluation plan. 

.. More small groups scheduled. 

. . Assignment of tasks to be carried out by sub-groups among the 
participants. 

. . A background in statistics, 

. . More time for more detailed lectures by the experts in the field. 
.. Better distribution of lectures and work sessions. 

. . More small group interaction — groups changing rather than static. 
.. More time actually organizing for activities. 

.. More practice in developing instruments. Sharing ideas for 
these instruments . 

.. Better meeting room — work room. 

Which one of these phrases best states how related this workshop 
was to your interests and background? 



a. It was over my head 0 

b. I understood almost everything but the 

conference missed my main interests 2 

c. It dealt with my main interests in an 

understandable and interesting way 19 



d. It was too basic , few if any new ideas .... 0 

Which of the following statments comes closest to stating your 
general reaction to the total Workshop? 



a. The most valuable educational experience of my life 0 

b. An outstanding program, I received much from it 15 

c. Many parts were valuable, others not very 5 

d. I gained something from attending but less than I expected . . 1 

e. It was almost a complete waste of time 0 

f. (Other), 0 



After this Workshop is over, is there anything related to the 
Workshop topics that you would like to know more about or to 
study further? 



a. Yes 19 

b. No . . 1 

c. No Response. » . , . , 1 

(If yes) What specifically would you like to study? 

.. Statistics. 5 

.. Stake's evaluation model 4 

.. Interviewing, questionnaire development 4 

.. Developing rating scales 1 

.. Flander’s method 1 

. . Bibliography for future reading . . . 1 

.."I'd like to know as much about evaluation 

as my one cell can tolerate." 1 
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10. (If yes) How would you like to do s /l 



a. Study on my own. . , . , 

b. Attend a class that meets weekly , 

c. Attend another Workshop . . .. . 

d. Take a course by correspondence. . 

e. In a local study group 

f • (Other ) c « * i. ..>«,.««» 

No response, 



Combinations of the above 

( 3 ) and (b) ♦ 1 

(a) and (c) * . , 5 

(b) and (,c) 2 

(c) and (p ) ........... 1 



3 

0 

3 

1 

0 

2 

3 



If you have further comments on the Workshop, please write them 
in your own words . 

I would feel somewhat lost if there were no follow-up. 



I think if I had had enough time to read some of the recommended 
bibliography before, I would have better understood the lectures, 
especially terms used. 



Rapid pace might profitably be reduced by extending it over an 
additional week. 



Really appreciated freedom to question, discuss and choose areas 
of study. 



A number of participants felt that the staff presupposed that they had 
more background in statistics and mathematics than they did. In fact, some 
felt that more time should have been devoted to statistics than some other 
topics tnat were considered. 



Participants* reactions to environment and facilities, scheduling and 
organization, content and presentation are reflected in the following 
tabulations and commentary „ 



Environment a nd facilities 



1. (a) To what extent did the relative unavailability of books 

and journals interfere with your attempts to master the 
content of this session? 

No interferences 14. Little interference: 5. Interference: 5. 



o 
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Those who indicated "no interference’.' stated that the 
institute’s concentrated program precluded reading time and 
that leading after rather than during the session would 
be more meaningful One person had brought some recommended 
textbooks for reference during the institute. 

Those who indicated "interference" suggested that available 
books would have stimulated reading or that references cited 
at the institute should have been available. One person 
wanted to use some statistical tables and read some articles 
that had not been provided. 

(b) To what extent did reproduced materials given to you by the 
staff improve matters? 

Most responses indicated that this material helped a great, 
deal. Two indicated that more articles should have been 
provided. 

2. (a) Did you feel that you lacked a "place to work," either alone or 

or in small groups? 

All but two responded "no" ( . One indicated the iited for a 
work room equipped with typewriter, reference materials, etc. 

(b) If you had a room at the Union, was it satisfactory? 

All living at the Union responded "yes". 

(c) If you did not have a room at the Union, did your staying 
elsewhere make the institute any more or less worthwhile 
to you? 

All but three responded that housing made no difference. 

Two considered it a disadvantage. One considered it 
advantageous. 

3. (a) Which features of the meeting rooms were inadequate or not 

conducive to learning? 

Lack of table and writing surfaces, inadequate blackboard 
space, distraction from adjacent rooms, handling of equipment. 

(b) Which features were especially facilitative in the same regard? 

Central location, number of available rooms, air-conditioning, 
"opportunity to work with Dr. Stake's graduate students". 
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Scheduling and Urbanizatio n 



4. (a) Was two weeks too long a period to leave your work at home 

for the purpose of attending this session? 

Twenty-two said no; the rest said yes but stated it was 
worth it. 

(b) Was two weeks too short a period in which to learn much 
of the content of this session? 

Twelve of the 24 respondents said "yes”; one did not 
answer; eleven said "no" (two qualified their "no" with 
the exception of statistics). 

5. (a) Were you allowed enough time in which to pursue activities 

of your own choosing? 

Yes: 18. No: 4. 

Two respondents qualified their answers. One said "no" to 
the first week and "yes" to the second week. The other stated 
that the activities of the second week were not described 
enough. "There should have been more of a push to get 
somethin g developed on a tentative basis and turned in after 
the first week." 

(b) Would you have preferred not to meet in the evening after dinner? 
Yes: 8. No; 3. 

The remainder, for the most part said "yes" and "no", "sometimes", 
"didn't mind". One said "no" to lectures in the evening 
but added that other activities were fine. 

(c) Would fewer meetings- per day have been preferable? 

Yes: 3. No: 14. 

The rest of the. participants felt it was "dependent upon content", 
"sometimes" it would be preferable, "an individual matter", 
"dependent upon lecture time as work session time." A few 
said "sometime" or "not always" and one did not respond. 

(d) Would you have preferred more meetings per day than there 
actually were? 

Yes: 3. No: 16. No response: 1. 

One said 4 meetings per day is enough; another said "on 
certain days"; one stated "sometimes" and one stated "it depend- 
ed upon lecture time vs. work session time." 
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6. (a) Were the individual lectures too long to sit and listen or 

take notes? 

Yes: 2. No: 12. No response: 9. 

One respondent said the lectures were too long when two 
lectures were consecutive. 

(b) Were the lectures scheduled in an appropriate sequence? 

Yes: 13. "Usually", "not always", "Sometimes": 5. 

No response: 3. 

Other responses: 

"In regard to information, seemed logical; to assignment, no." 
"Hastings should have been scheduled earlier when we still 
were working on plain." 

"Statistics earl ier-measures- then model-f inally develop 
individual plan." 

7. (a) Did you have sufficient opportunities to interact with 

other participants? 

Yes: 13. No: 8. "Not always": 3. 

8. (a) Were the instructors too inaccessible or unapproachable so 

that you did not get the individual attention that you desired? 

Yes: 2. No: 14. "Sometimes," "Not usually," or "to a 
slight degree'.’: 7. 

One participant commented, "If we could have had the coffee 
break together it might have been easier to express my ideas 
in a more relaxed atmosphere." 

One respondent stated that the instructors were very approach- 
able, although they could have been available more during 
work sessions. 

(b) Would it have been advisable to have had a few highly-trained 
graduate student assistants present from whom you could have 
obtained help on individual problems? 

Yes: 10. No: 7. "Possible": 4. No reactions: 3. 

(c) Were the staff members helpful in any way? 

Yes: 21. Most or some: 2. No reactions: 1. 

9. (a) Did the attempts to evaluate your progress and reactions 

during the session (and at this moment) interfere with 
your work here? 

Yes: 2. No: 22. 
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(b) Do you begrudge the time you have spent here answering such 
questions as these on this critique? 



Yes: 0. No: 23. "Weill": 1. 

10. In general, was the institute well organized? 

All said "yes". One qualified the answer by stating there 
was poor scheduling. One individual said "One of the best 
institutes I have attended"; another, "Better than previous 
ones attended." 

Content and Presentation 



o 



11. (a) Did the content of the lectures and readings presuppose far 

more previous training (in math and statistics) than you had? 

Yes: 7. No: 11. "Perhaps at times": 6. 

(b) Should less training in these areas or more have been presupposed? 
(This question was unclear to participants.) 

12. To what extent was the content of the lectures and readings 
relevant to what you hoped to accomplish during the session? 

Many misinterpreted this question;, some indicated some 
relevance. 

13. Do not be reluctant to single out a staff member for praise 
or censure. 

Denny: 7. Stake: 4. Ernie: 3. All contributed: 9. 

Sjogren: 1. No response: 2. 

13. (a) Were the lecturers stimulating and interesting? 

Yes: 15. "Usually", "Most": 7. Some: 1. ?: 1. 

(b) Were the lecturers competent to speak on the subject assigned 
them? 

All stated "yes". 

(c) Were the lecturers well prepared? 

Yes: 21. No: 0. Some, not: 3. 

14. Were you disappointed in any way with the group of participants? 
"No!": 15. 
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Others commented: 

’’Seemed to be a left skewed distribution of participants, 
a few were rather uncommunicative." 

"At first I felt the group should have been more homogenous 
in background — I believe now that the heterogenity of the 
group was a positive influence." 

"Too many varying backgrounds." 

"I thought there would be more classroom teachers; not enough 
classroom teachers." 

One participant did not respond. 

Answer each of the following only by checking the more appropriate blank: 

15. If you had it to do over again would you apply for this 
institute which you have just completed? Yes: 24. No: 0. 

16. If an institute such as this is held again would you recommend 
to others like you that they attend? Yes: 24. No: 0. 

17. Do you anticipate maintaining some sort of contact with 

at least one member of the institute staff? Yes: 22. No* 1. 

No response: 0. 

18. Do you feel that your understanding of evaluation has been 
considerably enriched in these two weeks? Yes: 24. 

19. Is it likely that you will consult in evaluation with someone 
else attending this institute? Yes: 22. No: 2. 

A few punctuated the "yes" with a question mark or "Hope so." 

20. Would you say that because of this institute you are more 
able to state a given evaluation problem in operational 
form so that it is, if it can be, amenable to solution? 

Yes: 24. No: 0. 

21. Do you feel that the staff should feel that it has accomplish- 
ed its objectives during this two week institute? Yes: 23. ?: 1. 
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Interviews 



In order to elicit participants’ perceptions about lectures, materials, video 
tapes and the overall institute design (organization and procedures) and 
to gather data about the participants' potential application of these 
learning variables in their own situations, interviews were conducted. 
Interviewers asked how beneficial and helpful the institute had been, 
whether the components had been convenient or available, if the experiences 
had been understandable. 



At the end of the first week, half the participants (selected at random) 
were interviewed; the other participants were interviewed at the close of 
the second week. After the initial interviews, suggestions for altering 
the second week were incorporated in the presentation. From all che 
interviews, suggestions for altering the program were culled for future 
reference. 

In the following itemization of participants' suggestions and the staff's 
observations (informally made after the institute), individual suggestions 
reflect only personal rather than the group's views. 

. . . Short term suggestions 

"Review the next day's schedule and purpose./' 

Observation: This was done only once. 

"Explain other activities and agencies such as CERLI and those on the 
Urbana campus and include what is happening now." 

Observation: In lieu of taking institute time for this, it was suggested 

that interested participants write to CERLI. 

"Not sure which of staff will be helpful-need background data about staff." 

Observation: Some but unsuccessful attempt was made to alleviate this 

situation. One lecturer declined to give information. 

"Clarify following-through plans; purpose and meetings." 

Observation: In response to questions, the participants expressed the need 

for follow-up activity- preferably in task-oriented groups rather than a 
general meeting. The CERLI staff indicated that future planning and meetings 
might take place at the laboratory in Northfield. 
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"Need more free time to study." 

Observation: During the first week, no formal assignments were made. 

During the second week, no evening sessions were held; but this week seemed 
almost anticlimatic . 

Participants— staff relationships i.e. coffee together at break, evening 
parties, etc." 



Observation: During the second week, coffee was served. During the first 

week, a party was given; none was given the second week. 

Help people more with statistics: get them to work on it, try it." 

Observation: Not much done about this suggestion. 

Bind everyone s evaluation plan for critique and future reference." 

Observation: The plans will be circulated. 

"Need more copies of Mirrors of Behavior ." 

Observation: During lectures, this text was cited but apparently only four 

participants used it during the institute (although it might have been 
circulated without being checked out). Observers suggested that lecturers 
identify relevant references for participants to read before attending the 
scheduled lecture. 

"Statistic tables, square root tables, etc. needed." 

Observation: Not provided. 

"Role playing for practice and to get to know each other." 

Observation: Not cone. 



"Small group reaction to tasks (?) material, and evaluation problem." 
Observation: Nothing done. 

"Staff members as facilitators." 

Observation: For the most part, Dr. House’s staff felt that their lack of 

training in small group leadership precluded this function. 

"Need time (five minutes or so) to think and ask questions." 

Observation: Lecturers were not told this although they should have been. 

More interaction between lecturers and participants should have taken place. 
More rest periods should have been scheduled. 
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Show more alternative uses of models and techniques." 

Observation; Sorenson was referred to but his ideas were not developed, 

"More discussion of what the video tapes show and more discussion after their 
viewing. Make more tapes accessible to individuals for viewing." 

Observation: Nothing done the second week although the tapes were shown 

hourly. 



, , . Long term surest ions 

The following participants’ suggestions and criticisms offer specific comments 
for planning future institutions- particularly the recurring points that reflect 
mutual comments. Like the short term suggestions, however, many statements 
reflect individual rather than group opinion. 

Orientation. . . 



More small group work on expectations. 

Want to get to know people-meet participants. 

Some kind of overviewing beforehand. 

Procedures and Descriptions , . . 

Provide clearer picture of participant role following the institute. 
Follow-through plan. 

More staff time with participants. 

Better meeting facilities. 

Pay more attention to what threatens participants. 

Some simulation and problem approaches. Include role playing. 

Two weeks not long enough: Let participants have more time for study 

and formulating evaluation plans. 

More grouping for instruction. 

Build in even more options. 

Point out need for participants to have a high tolerance for ambiguity. 
Give participants a chance to critique each others’ plans. 

More work sessions the first week. 

More opportunity to talk with experts. 

More variation in scheduling-lectures too long. 

There were too many work sessions in a row and too many lectures in a row. 
More time on interviewing, role playing, etc. 

Should have done a pre and post role playing. 

Chance to meet others and see progress (using video tapes). 

Lack of communication between participants and staff. 
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Force participants to work through statistical problems with staff. 

Terminology used by lecturers often confusing. 

Some participants felt that secretarial service, typewriters, prepared 
lecture guides, sample scales and questionnaires, statistical tables 
and mo- e . * ..t • :*. + . icni problems should be provided. 

Hastings was too difficult. 

The sequence of lectures not satisfactory. 

The ERIC visit was of no value. 

There was too much assumed about participant background. 

The participants were split on statistics-some knew it; others, not. 

Some of the participants felt that they would n t be able to use parts 
of the training because: (1) they had no access to control groups, 

(2) statistics would be done by someone else (3) no video taping 
equipment available, (4) the teachers were not sure of the potential 
use of the institute. 

Lectures too classroomy. 

Participation should be limited to administrators. 

There should be more on student behavior. 

Participants should be more homogeneous. 

ur. .louse's staff should have been more involved. 

Integrate a developed instrument of the second week in the evaluation 
plan of the first week. 

Materials and Video Tapes . . . 

Have a large definite reading list prior to institute. 

Send materials earlier. 

Edit tapes — too long. Describe purpose. 

Show some relationships between tapes and evaluation plans. 

More concentration on understanding the statistical exercises. 

Work on evaluation plans — some should be programmed as opposed to class 
oriented. 

Closer look at tests, scales, and other measures. 

Plan to bind • .’l evaluation plans. 

Develop a progression tape showing levels of planning. 

The tapes needed editing and a greater variety of interviews peeded to 
be shown. Some felt more personal interaction with Stake would 
have been more beneficial than watching Stake on video. 

There seemed to be a big gap between the video tapes and the written 
plans. The problem of the tapes did not seem related. 

Positive and General Reactions to Total Experience . . . 

Lectures are good, relevant, interesting. 

Would have come even if there was no stipend. 

Been excellent — great. 

Video tapes excellent learning device. 

Materials just what needed. 

Unobtrusive measures- things even teachers can do. 

Caliber of staff — stimulated. 

Lectures pertinent — understandable. 



Practical — useful video tapes. 

Evaluation plans model for writing* 

Lecturers don H t talk above us^ Good information. 

Different approach to evaluation. That’s good. 

Statistical exercises, practical, easy. 

Stake alluding to community attitudes. 

Ways at getting at information. 

Speakers cover all areas of gifted programs. 

Evaluation plans served as models. 

Sharing of ideas, hearing about what other programs are doing, anything 
that was considered as input in designing evaluation, pre and post 
idea, these were considered (other) useful activities. 

Personal Opinions . . . 

Relaxing of evaluation philosophy. Greater picture. 

Staff provides help and service, not telling how much they know. 
Non-religious attitude toward evaluation. 

Good to hear authors of concepts. 

To get examples of what is in concept papers. 

Like to hear what authorities think should be done and questions 
to be answered* 

Staff work with participants 7 problems. 

Needed most of the information. 

Like Staff's attitude toward evaluation. 

Don’t feel (as once did) that evaluation is impossible. 

Dynamic personalities. 

Models good — appropriate., 

Stake’s model and lecture were considered by the first group of 
participants to be of the most benefit. 



Evaluation Plans 



At the conclusion of week one, participants submitted their plans which a 
staff member read, annotated and rated. Since only a few participants 
previously had attempted any kind of evaluation, the "ratings" of the group’s 
plans would seem to indicate a worthwhile effort: 10 "good", 9 "adequate", 

3 "inadequate". 

Among the "good" plans, the following have been selected for publication 
in this report. 



EVALUATION PLAN FOR GIFTED PROGRAM 
Lombard, Illinois 

Problem : Can a more positive attitude be created in which the program 

can operate? 



School situation : 

The independent study group consisted of nine fifth and sixth grade 
students. They were selected from six classrooms consisting of 
approximately 160 students. The selection was made according to the 
results of the Otis Quick-Scoring Mental Ability Tests, the Stanford 
Achievement Tests, the Torrance Tests of Creative Ability, and to a lesser 
degree on the judgment and recommendations of previous teachers. 

In general, tne children selected to be given the Torrance Test were 
those whose intelligence quotients were 120 or above and who were 
achieving at or above the 95th percentile on the Stanford Test. The 
Torrance Test was then administered and from this data, along with teacher 
recommendations, students were selected who scored highest on an over— all 
basis. 



Problem situation : 

Several students in the group had mentioned that they felt they were the 
most intelligent children in the entire school; that they were the "smartest, 
the obvious leaders." This was not mentioned to the director, but rather 
discussed among the students as they worked. 




- 45 - 



On occasion the students did mention to the director that they were 
missing some event or examination in their regular classroom. They seemed 
to be torn, at such times, as to whether they would rather remain in the 
regular classroom or attend the independent study group. 



Evaluation Plan ; 

Rationale : 

Those administrators and staff members in the district who are involved 
with the gifted program feel that talented youngsters will benefit from 
the opportunity to work independently on topics of their own choosing. 

However, it is also felt that the above mentioned negative aspects of the 
program can be detrimental to the youngsters, as can the conflict in 
loyalties (being forced to choose between the regular classroom 
teacher and the director of the independent study group). Furthermore, 
it is not our intention to develop an elite group or a group that considers 
itself to be elite. 

Purpose : 

The purpose of this evaluation is to ascertain the attitudes of the 
children in the group concerning their participation in the independent 
study program and the attitudes also of the othrr children in the fifth 
and sixth grades toward the participants. Also the attitudes of the 
other staff members must be considered. 

The children involved in the program seemed to feel set apart from the 
other children. On occasion they were not included in such special activities 
as field trips, opportunities to hear guest speakers, or the viewing of 
films, filmstrips, or plays, as these activities were scheduled on the 
half day they were out of the room. They also were often out of the 
regular classroom when new skills or concepts were presented or when 
examinations were administered. 

Some of the children had the impression that they were intellectually 
superior to all other children in the school, since they had been 
selected to be included in the program. They expressed the belief that 
they were entitled to special privileges (election — automatically — as 
class officers, highest grades in the regular classroom, special field 
trips, equipments and materials not made available to the other children) 
because of this supposed "superiority." 



Procedures : 

A. The following procedures will be used as a guide to measure the students* 
attitudes regarding their participation in the independent study group. 

1 Administer sociograms in the fall to all fifth and sixth grade 
students before selection is made regarding participants in the 
independent study program^ This sociogram will determine: (a) who 

the children feel are the room leaders, (b) who are their best friends. 
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and (c) who they would least like to sit beside. Readminister these 
sociograms again at mid-year and at the end of the year to assist 
in locating changes in relationships, 

2. Interviews with members of the selected group conducted by the 
school psychologist in the fall and periodically throughout the 
year to ascertain their entering attitudes and any changes in 
attitudes, 

3. Hold general group discussions to vent their feelings. 

B. The following procedures will be used to measure the attitudes of the 
other fifth and s ixth grade students toward the participants in the 
independent study program, 

1. Administer the previously mentioned sociogram at the three suggested 
times to determine changes in friendships as a result of participa- 
tion in the program. 

2. Meet with these children and explain to them the whole scope of the 
program so that they will better understand what their classmates 
are involved with when they are out of the classroom. 

C. The following procedures will be used to assist in the interaction 
with fellow staff members and the community. 

1. Invite the teachers to periodically visit the independent study 
program to observe what is taking place. 

2. Report frequently to staff members regarding the progress of the 
program as a whole. Perhaps this could be worked into building 
meetings on a once-a-month basis. 

3. Confer on an individual basis with the fifth and sixth grade 
teachers whose students are involved in the program reporting 
on the work of those students. 

4. Confer with the principal regarding any feedback he may receive 
from other teachers or from the community. 

5. Try to obtain feedback from various parents in the school community. 

6. Prepare a list of suggestions or recommendations regarding classroom 
activities on the half-day the children are out of the room. (Examples: 
avoid special activities such as guest speakers, etc., avoid 
examinations on these days , and refrain from the presentation of 

new skills and concepts during this half-day period of time.) 



ERIC 
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EVALUATION PLAN FOR DEMONSTRATION CLASSES FOR GIFTED CHILDREN 

Marion , Illinois 

-«•> ■ 

Problem : 

Does the demonstration program for gifted children in our school 
increase the students ability to think divergently? 



School Situation : 

In this junior high school of approximately 900 students, students from 

£he unit elementary schools are identified in the sixth grade for the 
gifted classes in English, social studies, and science through a three-fold 
testing program including the SRA Achievement Test, the Otis-Lennon Test of 
Mental Maturity, and The Torrance Test of Creative Thinking (Form B) . 

Scores on these tests (or, in the case of the SRA tests, scores of those 
areas for which identification is taking place) are converted to stanines 
ranging from 1 to 9. The stanines for the three criteria are added, so that 
total stanine scores for any individual student could range from 3 to 27. 

Those students (approximately 30 per subject area) with the highest stanine 
averages are then placed in the demonstration classes. Since identification 
procedures distinguish between content abilities, any individual child may 
be in only one or all three special classes. 

Students who are identified for the junior high program are normally 
expected to remain in the program for three years. Rarely is a student who 
was not originally identified allowed to enter one of the programs after the 
seventh grade (especially in English and social studies) since these studies 
are highly sequential as well as inductively presented. 

With only minor variations in the selection criteria described above, the 
demonstration program has been in operation in English and social studies 
for four years and in science for one year. The English classes use the 
Project English curriculum developed at the University of Illinois under the 
direction of Dr. James McGrimmon. The social studies classes use the Project 
Social Studies Curriculum also developed at the University of Illinois under 
the leadership of Dr, Ella Leppert, The science classes use accelerated content 
and emphasize a laboratory approach to learning. 

Problem Situation : 

The teachers in the demonstration classes are aware that the identifica- 
tion procedures used are designed, at least in part, to select students who 
are not simply "bright” in the sense that they achieve at a high level but 
who also have the ability to think more creatively. The materials used were 
specifically designed to encourage discovery learning and to stimulate 
divergent-evaluative thinking as described by J. P. Guilford. This being 
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the case, one question which interests the teachers is whether the program does, 
in fact, increase the students’ ability to think divergently. 

The Director of the Demonstration Center agrees that the question should 
be examined. He has agreed, therefore, to provide whatever data he possesses 
and materials for such an evaluation. The guidance person who has been assigned 
to the gifted program will administer various tests, and the Demonstration 
Center secretary will provide clerical services. 

Evaluation Plan 



Rationale : 

The gifted program in this school is committed to a great extent to the 
philosophy that in a complex society such as ours, factual knowledge, per se , 
is of less benefit in producing citizens who can function effectively in their 
environment than is process knowledge. That is, that those individuals who 
are best able to cope with complex situations are not those who think primarily 
on cognitive -convergent levels, but those who think on divergent-evaluative 
levels as well. 

Consequently, the school has provided homogeneous classes for those students 
who have been identified as most likely to benefit from a program designed 
in part to encourage divergency. 

Purpose : \ 

The primary purpose of this evaluation is to determine whether students 
who participate in the demonstration classes do increase their ability 
to think divergently, 

A second, somewhat parallel purpose is to attempt to identify specific 
aspects of the program which tend to enhance or inhibit the development 
of the students’ ability to think divergently. 

Procedures : 

I. Pre-and-Post Tests 

A. Torrance Test of Creative Thinking 

Since all students in the program have taken the Torrance Test 
of ''reative Thinking prior to their inclusion in the gifted 
program^ a re-test of that same test will be administered at the 
end of each academic year. Since the thought skills demanded on 
this test are primarily divergent, a change in the students' 
composite performance scores would indicate a change in their 
ability to think divergently. 

Perhaps as an attempt at control, a random sample of other 
children in the school could also have the re-test administered 
to them, (All children in the school unit were originally tested.) 
Although these children are not comparable to the gifted class 
in many ways, an increased ability to perform tasks requiring 



-49 



divergent thinking equal to the increased ability of the students 
placed in the gifted program would invalidate the idea that the 
gifted program rather than maturity was responsible for increasing 
this particular ability. 

By administering the tests at the end of each year (or perhaps 
the alternate form), periods of greatest enhancement or inhibi- 
tion should be readily observable. The design of the materials 
or experiences in those time periods which show greatest enhance- 
ment or inhibition could then be analyzed carefully for suggested 
changes in the curriculum, 

B, Guilford Consequences Test 

The Guilford Consequences Test might also be given to the in- 
coming seventh graders at the beginning of the year and then 
again at the end of either each academic year or at the end of the 
three year program to be used much as the Torrance Test of Creative 
Thinking will be used, 

II. Classroom Observation Devices 

Several classroom observation systems will be used at frequent 
intervals in each of the gifted classes in order to assess exactly 
what kind of behavior is occurring in the classroom, what kind of 
atmosphere is generated, and what kinds of thinking the teacher is asking 
the students to do. This measure will be compared to ideals set up 
by the demonstration teachers. For example, the teachers feel that 
in order to stimulate divergent thinking on the part of the students, 
a teacher should aim for a non-authoritarian, friendly manner and 
that he should teach concepts inductively. If the observation tools, 
however, reveal that generally the teacher is authoritarian and teaches 
deductively, then this data would be of significance in analyzing the 
degree of change (or lack of it) in the students’ ability to think 
divergently as a direct result of our program. 

In order to examine many facets of the classroom situation, 
all of the following observation systems may be implemented through 
the use of personal observation, audio tape recordings, and video 
tape recordings: 

A, The Flanders Interaction Analysis 

B. The Denny Observation Scale 

C, The Galloway Non-verbal Interaction Analysis 

D. The CERLI Behavior Analysis 

III. Interviews 

At various points in their academic careers, the students will 
be interviewed individually and in small groups by various personnel 
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such as state consultants, guidance personnel, and teachers. Carefully 
designed interview formats will be followed to elicit information 
regarding an increased ability to think divergently. If possible, 
some interviews might be conducted even after the students have left 
the junior high school setting. For example, the hindsight afforded 
high school seniors or college students who had been in the program 
might provide significant information. 

IV. Unobtrusive Measures 

In addition to the more formal procedures outlined above, 
teachers will be asked to take note through the year of information 
gleaned from casual conversations with students, with other teachers, 
and with parents. Those teachers responsible for sponsorship of various 
student organizations such as Student Council, the science club, or 
the art club will be asked to keep a record (rather informally) of 
ideas and contributions the gifted students make to the organizations 
to which they belong as possibly bearing on the individual’s ability 
to think divergently. 



EVALUATION PLAN FOR TABA SOCIAL STUDIES CURRICULUM 
Park Forest Public Schools 

Problem: Are teachers using the Taba Social Studies Curriculum changing their 

behavior in such a way that the gifted students in their classes 
are able to carry out the development of generalizations, i.e. 
interpret data and make inferences, at a higher level than those 
developed by less able students? 



School Situation : Two years ago two of the nine elementary schools in Park 

Forest piloted the use of the Taba Social Studies Curriculum in their 
classes from grades one through six. Prior to the introduction 
of these materials a group of eleven teachers and administrators attended 
a one-week workshop conducted by Dr. Taba and her staff at San 
Francisco State Collegr 

Following the first year of use, another group consisting of some of 
the original group plus several additional teachers attended a second 
workshop at San Francisco, During the second year of the program 
three additional schools began their use of the materials. Through- 
out the year, in-service activities were carried on under the 
directi', n of the administrator most involved in the program. 
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This summer the administrator responsible for the in-service aspects 
of the program will receive additional training at a two-week 
workshop and will have available for use during the year a packet 
of materials prepared under the direction of the Taba Project for 
the in-service program., 

In the school year : 68- f 69 ? two additional schools will begin 
using the program for the first time. The two schools not in the 
program will be used when a control group is called for. 

Classes in all of these buildings are heterogeneously grouped 
and average slightly under thirty students per class. Approximately 
eighty teachers will be involved in teaching these materials of which 
about twenty five will be teaching the program for the first time. 

Gifted students are identified but not separated for any classes. 

With the exception of music the classrooms are self-contained units. 

Problem Situation? The State of Illinois Gifted Program has accepted the 

premise of the program; that is gifted students can be provided for 
in heterogeneously grouped social studies classrooms. Verification 
by means of an evaluation of this premise has never been accomplished. 

According to Dr, Taba the program will be successful only if the teachers 
will follow certain generic teaching strategies which apply to any 
topic and practically any type of teaching situation. For the 
purpose of this evaluation, only the teaching strategies related 
to the development of generalizations will be studied. 

Evaluation Plan : The following evaluation plan, developed after consideration 

of many aspects of the problem, should be considered as preliminary 
and subject to modification before being put into effect. 

Rationale : 

From the time that the Park Forest Schools first came into existence, 
less than twenty years ago, a basic philosophy of the district has 
been to provide for * "dividual differences among all students. 

Through the years many methods have been tried, many materials used, 
a variety of teaching strategies employed in an effort to better 
reach this goal. 

Along with this goal, the philosophy has included another aspect — 
that of working within self-contained classrooms with heterogeneously 
formed groups,. The intellectually talented and the creative child 
are identified, even though some of the identification methods are 
open to question; but they are not placed into separate programs or 
classes. 
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Purpose 

A primary purpose of this evaluation is to ascertain whether teachers 
using the Taba Social Studies Curriculum can and are using the 
planned teaching strategies necessary for the success of the program. 

If teachers are successful in using the strategies designed by Dr. 

Taba. the assumption is then made by Dr. Taba that these strategies 
will result in changes in student behavior, e.g. the ability to develop 
generalizations. The secondary purpose of the evaluation then becomes 
a matter of trying to determine if students do in fact develop the 
ability to make generalizations. 

The third purpose is to determine if gifted students, as is claimed 
by Taba, develop generalizations in greater depth than do average 
or less able students. 

Procedures 

For the purpose of chis evaluation the following procedures are 
proposed: 

Teacher evaluation; 

1. The Taba. System of Classroom Observation will be used on a 
randomly selected sample of teachers using the Taba materials for 
the first time and on teachers in the two schools not in the program. 
These observations will be made at the beginning of the school 

year during social studies classes and will consist of five minute 
segments recorded at three different times. 

One coder will work with these observations and from the data collect- 
ed will draw up a profile of each teachers performance. During 
the last month of school this same procedure will be followed and a 
second profile prepared. 

Three judges will look, at these profiles independently and judge 
whether or not, in their opinion, the teacher had made significant 
changes in their performances in reference to their ability to help 
students make generalizations • These results will be compared and 
a final report on this aspect of the evaluation prepared. 

2. A brief survey will be prepared for the Taba teachers to determine 
the following: 

a. Teacher feelings as to whether or not they had changed as a 
result of being involved in the in-service program for the Taba 
Social Studies Curriculum. 

b. Teacher feelings as to whether or not they were better able to 
take into account individual differences in their classes. 

c. Teacher feeling as to whether or not the program provided 
gifted students an opportunity to develop cognitive thinking skills. 



3 o The Teachers [ Handbook prepared by Dr, Taba lists and explains 
five teaching strategies for her program. Video tapes will be 
made of a randomly selected group of Taba teachers at two-month 
intervals. The teaching strategies as seen in these tapes will be 
compared to the ideal (as specified by Taba) by a panel and rankings 
will be prepared. 

Student evaluation - 

1 . All students from randomly selected classes in the program and 
classes not in the program will be given a collection of data and 
asked to prepare a report on the basis of this information. A panel 
of three junior high social studies teachers not involved in the 
program will be asked to rank these reports on the students’ ability 
to make inferences from the data. 

Comparisons of these results will be made between control and 
experimental groups and between students identified as gifted and 
the less able students within the experimental classes. 

2. An unobtrusive measure of the number and types of books used 
by students in the experimental classes will be taken. Since the 
books used in the program differ in complexity and type, it is hoped 
that this measure will give some clue as to whether or not the 
gifted students use more complex books and books in greater 
numbers in developing generalizations. 

It is possible that the following unpublished documents by members 
of the Taba Project will yield other useful measures. 

"Test Exercises in Listing, Grouping, Labeling and Generalizing 
Skills", A. H, McNaughton 

"Informal Tests on Concepts and Generalizations", A. H. McNaughton 
"informal Evaluation of the Ability to Interpret Data in Written 
Materials and Make Inferences from Them", A. H. McNaughton 
"A Generalization is a Generalization", A. H. McNaughton 

On the basis of all the above listed evaluation procedures, a final 
report will be prepared. 



EVALUATION PLAN FOR INDEPENDENT STUDY 



RATIONALE: 

The academically talented child readily learns and retains information. 

By his very nature he constantly is seeking new challenges to his thinking 
and creativity,, Often, however,, there is limited opportunity for him to 
apply the skills and information he has acquired in a way which is meaning- 
ful and rewarding to him. Thus, the need to establish an atmosphere which 
allows the child the freedom of alternatives and the opportunity for self- 
initiated activities becomes mandatory. With these facts in mind, the traditional 



concept of the school's role in directing the student* s efforts must be 
modified. 

We are, therefore, faced with determining the kind of educational environ- 
ment that will allow our intellectually superior children to use their 
talents to the greatest efficiency so that they can grow up to be effective, 
contributing members of our society, 

INTENDED ANTECEDENT: 

This project will involve an experimental and control group of intellectually 
superior sixth graders from the Holmes Junior High School. The students have 
been identified according to their scores on Large Thorndike Intelligence. 

The experimental and control groups will consist of ten students per group. 
They will be randomly selected from the top ten per cent of the gifted class 
as determined by their IQ score,- The final determinant will be their ex- 
pressed willingness to participate in the experimental study and parental 
support of the undertaking, 

INTENDED TRANSACTIONS: 

This project will extend over a three year period beginning in September, 

1968 and terminating in June, 1971. 

The students selected for participation in the experimental group will be 
scheduled into accelerated classes as will those in the control group. 
Objective data, not included in the identification program, will be gathered 
at this time. In addition, data of a subjective nature will be collected. 
Such things as attitude toward school teachers, classes, classmates and 
assignments will be noted along with noticeable strengths and weaknesses in 
self-initiative, self-discipline and independence. 

During these first weeks, the Project Director will be establishing the 
‘oundwork for a smooth transition from the existing program to the modified 
, ugram for both students and classroom teachers. Interested teachers, 
jhe principal and the guidance counselor will be invited to serve with the 
Project Director in an advisory capacity. In addition, the Project Director 
will have contacted the parents of eligible students to explain the proposed 
program and enlist their support. 

After involvement in assigned classes for approximately one month, the 
experimental group will meet with the Project Director and the Interdis- 
ciplinary Team who will provide an orientation to the proposed program and 
indicate the range of options to which the student will be entitled. Every 
effort will be made to encourage a trusting relationship between the students 
and the Team, The student association with these staff members should not 
be threatening. The students should look upon them as resource people and 
guides in the process of educational decision-making. 




- 55 - 



After the orientation session, the experimental group will meet with the 
team to consider the following: 

1. The role of the school 

2. Educational objectives 

3 Diagnosed strengths and weaknesses 
4. Areas of interest and needs 

At this point, each child in the experimental group will be given the option 
to determine: 

1. Which classes to attend 

2. What assignments to pursue 

3. How to use time away from class 

4. Where to spend time away from class 

5. With whom to spend time away from class 

The Project Director will be available to assist the students and consult 
with the students at all times. Members of the Team will also be available 
upon request. 

As part of the developmental sequence of the program, the students' initial 
options will be very limited. The progress of each child in adjusting to 
these will determine the point at which additional options providing for 
greater independence will be introduced. 

INTENDED OUTCOMES: 

This program has been designed to provide the Elk Grove School System with 
data which will indicate the appropriateness of a program of this nature 
for the gifted junior high population. 

We hope to show that: 

1. Given the freedom in which to operate, the intellectually superior 
child will eventually become a more productive individual. 

2. He will learn to make educational decisions and evaluate the 
consequences for these decisions in an environment where guidance 
is readily available. 

3. Involvement in this program will not adversely affect achievement; 
instead, the child will apply his skills and information in a 

way which is more meaningful and rewarding to him. 

4. Teachers will re-evaluate their role with greater emphasis on the 
real needs of these students. 



EVALUATION PROCEDURE 
ROj X 0 2 

R0 1 °2 

This study will consist of two groups of ten children who will be selected 
on a random basis from the top ten per cent of the sixth grade class. 
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determined by their scores on the Lorge Thorndike Group Intelligence test. 

ox i 0 

ox 2 o 

0X0 

3 

0 X, 0 

4 

0X0 

5 

A pre and post test on the Iowa Test of Basic Skills will be administered 
at the beginning and end of the school year as a measure of achievement. 
Additional pre and post scores from the Lively, Grove, and Dempster Jr. 

High gifted programs will be analyzed to see if the program at the Holmes 
Jr. High has affected a change in achievement compared to other sixth grades 
in addition to determining any significant change between the two groups of 
ten in the experimental program. 

A measure of self-direction will be developed to aid in identifying those 
children who display characteristics which seem to their ability to perform 
in an independent study program. If time permits, Suchman, Inquiry 
Training and Critical Thinking will be administered during the year to 
determine whether a positive correlation between self direction and these 
measures exists. It is hoped that by gathering as much data as possible, 
we will know more about this illusive quality of self reliance, be better able 
to identify it and begin to know what will help children develop in this 
capacity. 



Cumulative record information will be gathered, conferences with parents, 
teachers and students will be ana tat ed, class attendance records will be 
checked to determine how often children who are given options of whether 
or not to attend class actually option not to. 

An independent Study questionnaire covering the major areas of Acceleration, 
Content, Resources'^ and Evaluation plus a questionnaire on Change in 
Behavior and Relative Effects of Technique Compared to other Techniques 
including the areas of Learning — Content, Learning— —Study Habits, Attitudes, 
and Social Interaction will be given to the group at the end of the school 
year to get an affective measure after one year of the program. 

A questionnaire to the community covering the full expanse of the educational 
programs will be developed to aid in understanding what the community 
views as important for their children and to determine how much understanding 
they do have in regard to the educational provisions of the district. 
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Ihe In-Service program for teachers will include a section devoted to 
evaluation as a ’’tool to understanding. " They will be introduced to some 
of the basic skills of test construction and interpretation and will work 
on developing a rating scale or checklist for use in their classroom. The 
teachers will be introduced to self-assessment techniques such as Style of 
Teaching Inventory and Flanders Interaction Analysis. The Flanders will 
be used as a tool for noting changes in t cache; directed— —student 
directed behavior as the school year progresses. Half-hour segments will 
be collected once a week on a random basis. The school's psychologist 
and the director of research and testing for the district will be an 
integral part of this section of In-Service. 

Upon collection of the data, appropriate statistical analysis will be 
done with the results aiding in determining the direction of the program. 



EVALUATION PLAN FOR 

ASSESSING INFLUENCE OF INSTITUTE PARTICIPANTS 
ON OTHER TEACHERS’ PRINCIPLES AND TECHNIQUES 



The Problem : 

To what extent have teachers attending the 1968 Summer Institute at 
the Eastern Illinois Development and Service Unit been able to influence 
other area teachers (2500) with respect to principles and techniques 
presented in the Summer Institute? 



Demographic Data : 

Seventy-two teachers attended Summer Institute 1968. These teachers 
represented 50 school districts in a ten-country area providing educational 
facilities for 30,000 youngsters. (See attached lists of participants, 
their school districts and subject material areas.) 

Problem Situation : 

Summer Institute participants have been administered three scales 
described as follows: 

1. Attribute Preference Inventory: 

This inventory was developed by John R. Hurley and Christie 
Randolf of Michigan State University to elicit information con- 
cerning how undergraduates in child psychology valued assorted behavior 
qualities in young children. We feel that the summer school 
participants will exhibit significant different ranking behavior on 



these attributes as a result of their participation in the summer 
school program. The instrument has been used in several studies as 
a measure of conventional vs. expressive. Comparisons will be made 
between a pre-and post-administration of the inventory. With this 
additional variable we plan to differentiate between male and female 
participant responses. In addition, an analysis of the differences 
between rankings by summer school participants, staff and student 
counselors (eight students employed to work with the summer institute 
program) . 

2. Style of Teaching Inventory: 

This instrument was originally designed for teachers to administer 
to their students to receive feedback information regarding specific 
teacher character as perceived by students. We are using this scale 
to analyze differences between ' : expected real" and "ideal" teaching 
behavior as perceived by the summer institute participant. The 
results of this scale will be compared with norms developed for the 
participants. This self-assessment instrument represents one of several 
which have been made availal \e for use by teachers throughout the 
summer program, 

3. Summer Institute Questionnaires 

The objectives of this questionnaire are to determine the relative 
expectations of participants with respect to the summer institute 
program and the utilization of concepts developed within local 
school districts during the coming year. Included are some survey 
questions to determine the needs for in-service education in the 
program development . The final report will include an analysis of 
data and information collected between forms A and B of this 
questionnaire , 

The significant problem facing us at this time is to develop a follow-up 
study which will include consultative services to these teachers as they 
implement "Plans of Action" at their respective school district. Copies of 
these plans are available. Tentative plans for individual follow through have 
been made with specific participants, and this information is included as 
reference material in this plan. 

Rationale : 

It is intended that a "critical mass" of teachers be trained in self- 
assessment techniques and the development of new teaching behaviors, and that 
furthermore s these teachers would influence other area teachers. It is assumed 
that through this "critical mass" education for the 30,000 youngsters in the 
EIDSU ten-county area will benefit from improved educational programs. 

Purpose : 

The purpose of this study is to gather information, analyze data and 
interpret results that relate to the degree to which this "critical mass" 
has developed and a significant number of area teachers have been influenced 
by summer institute participants. 
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Procedures ; 

In addition to the follow through visavis and the attached Plan of Action, 
the following considerations are to be made: 

1. Summer Institute Participants: 

a. Degree to which they have effectively implemented their Plan 
of Action as described. 

b. Information relative to teaching behavior through the use of 
the verbal interaction analysis matrix (CERLI), the. Style of 
Teaching Inventory, The Flander's Interaction Analysis System, 
and other tools used for self-assessment as employed in the 
Summer Institute. 

c. Attitudes and reactions of participants of the role of the 
Summer Institute in subsequent professional development and change 
teacher behavior. 

2. Area teachers other than Summer Institute participants: 

a. Teacher rating scales utilized on a random basis over a period 
of time — to be compared with norms developed on the data 
collected. 

b. Information related to attitudes of these teachers towards self- 
assessment and the implementation of new teaching behaviors. 

3. Local school district administrators: 

a. Degree to which Summer Institute participants are given the 
opportunity to conduct in-service programs in the local level. 

b. Role of the administrator in implementing the Plans of Action 
developed by Summer Institute participants. 
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IV. SUMMARY 



During a two-week institute, the training of 30 public school personnel as 
evaluators was undertaken. Among the participants, relatively few had any 
background in research and even fewer had previously conducted any evaluations. 

For the institute, a design and several materials and instruments were develop- 
ed. In content, the workshop evolved from evaluation planning to skill 
training in specific data-gathering techniques. 

Since all the trainees were directly or indirectly involved with the Illinois 
Gifted Program, the institute was planned to capitalize on their background 
by introducing and promoting good evaluation in the context of each participants 
immediate problem in his local situtation. Such a realistic approach afforded 
students the opportunity to work with real data, solve their immediate 
problems, share ideas with other participants and to simultaneously learn concepts 
and techniques of evaluation,, Thus, it was felt, participants would be more 
highly motivated and would more easily comprehend and retain concepts and 
techniques of evaluation. Apparently this approach was effective or so a 
short term analysis seems to indicate. 

Basically, a participant’s progress during the institute was assessed by 
examining the test instruments and reading the completed evaluation plans. 

Analysis of pre and post measures taken on the students indicates that 
significant shifts in attitude toward evaluation occurred during the institute. 
(This directional shift approximated attitudes held by an evaluation expert.) 
Students also made significant gains on the achievement test after the two- 
week experience. 



0 
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The fact that participants completed evaluation plans (submitted for review 
and circulation) that they actually can use in their local situations 
represents a meaningful measure of the institute's success. 

Adversely, however, problems of interaction, heterogeneity, and continued 
direction affected the institute. Feedback from the instrumentation 
underscored the problem of interaction: trainees consistently expressed need 

for interaction among participants and between experts and participants. 

The fact that both administrators and teachers were enrolled in the institute 
accounts for the problem of heterogeneity. Not only did administrators and 
teachers seem to have few mutual interests but each group related to the 
institute's content quite differently. Teachers experienced difficulty in 
relating to the material and recognizing its practical application in their 
local situation. Conversely, administrators found some of the material 
(statistics, for example) repetitious. 

The problem of continued direction developed during the second week of the 
institute. There seem to be two reasons why the participants' sense of 
purpose and feeling of satisfaction somewhat declined. For one thing, 
participants were not involved in nor committed to a specific task which the 
development of an evaluation plan had provided during the initial week. 
During the second week, some participants may have been frustrated by the 
necessity to make choices predicated on a knowledge of the entire field and 
identification of their own needs and priorities. For the conscientious 
but less knowledgeable participant, this activity generated problems. 
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Maj or Conclusions 




1. It is possible to train public school personnel in evaluation 
techniques within a short (two-week) period. 

2. The design and materials developed for this institute proved to 
be very effective. 

3. The institute’s weaknesses were a lack of personal interaction 
and, during the second week, an. inadequate sense of direction. 

(Better selection procedures and minor reorganization should 
alleviate these flaws.) 

The fact that participants' use of resource facilities was 
minimal may have been due to the necessarily time-consuming 
thrust of the two -week program. 

4. Long-range problems, including conditions w:? thin the local 
district that inhibit evaluation, must await follow-up meetings. 

5. Suggestions for improving the institute and materials are incorporat 
ed in this report 0 
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APPENDIX 



A. Letter 

B. .^rui.'iitire 

C. Application 

D. Observation Schedule A 

Observation Schedule B 

E. Attitude Inventory 

F. Achievement Quiz 

G. Participant Opinionaire 

H. Participant Critique Form 

I. Participant Interview Schedule 

( Parts 1 & II ) 
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COOPERATIVE EDUCATIONAL I «o w. frontage road 

I NORTHFIELD, ILLINOIS 60093 

RESEARCH LABORATORY, INC. | PHONE: (312) 273-2444 



Dear Sir : 

This summer we are conducting a two-week institute on evaluation* 

We have an excellent staff and expect to have some good training 
materials* We also hope we have a fresh approach. Although the 
evaluation institute is focused on the gifted, the techniques and 
materials are readily applicable to any educational program. We 
hope to involve personnel from Title III projects or other pro- 
grams. We are particularly, though not exclusively, interested 
in those with some research training. We think that this institute 
may enhance the evaluation of your project or school, program and 
invite someone from your district to apply. 



Sincerely, 




Ernest R. House 
Project Manager 
Project for the Evaluation of 
the Illinois Gifted Plan 
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APPLICATION FOR SUMMER WORKSHOP 
IN EVALUATION 

Date 

(Mr . ) 

NAME: (Mrs.) - 

(Miss) (Last) (First) (Initial) 

ADDRESS: 

PHONE: - 

SCHOOL DISTRICT: 

TYPE OF CENTER: Demonstration Reimbursement Experimental 

PRESENT POSITION: 

Please describe, briefly, your previous research training and experience: 



Please describe any evaluation efforts you are currently engaged in: 



Will you be in a position next year to do some kind of evaluation of 
your district's gifted program? 



i 

i 

i 
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PLEASE RETURN TO: 

Dr. Ernest R. House 
Project Manager 
Cooperative Educational 
Research Laboratory, Inc. 
P.0. Box 815 

Northfield, Illinois 60093 
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OBSERVATION SCHEDULE - A 



Speaker Date Lecture __ _____ Tape 

Scheduled Starting Time Actual Start Difference 

Scheduled Finishing Time Actual Finish Difference 

Staff in Attendance House Stake Denny Hastings Sjogren 

Number of Participants in Attendance: 



A. Observer f s rating of the speaker f s communication with the participants: 

1. speaker encourages questions _____ discourages questions ___ 

comment 

2. total number of questions asked 

3. speaker sensitive to audience reaction insensitive 

*4 .. 

comment; __ 

B. Rating of participants 1 questions and reactions: 

4. questions relevant _ questions not relevant 

comment 

5. questions insightful questions not insightful . 

comment 

participants comfortable participants not comfortable 

comment 

7 . participants bored interested enthusiastic 

comment 

C. Participants* attitudes toward instructional techniques: 

8* materials distributed materials not distributed 

9. materials relevant materials not relevant 

comment 

11 . audio-visual equipment used r^ot used 

11. equipment produced an effective presentation not effective 

comment 
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Participants* attitude towards presentation: 

12. lecture tape ...well prepared ; adequate ; not well prepared _ 

comment 

13. lecture tape ...presentation dull ; adequate ; interesting 

comment 

14. lecture tape ...presentation disjointed ; coherent 

comment 

15. lecture tape ...level of material difficult ; moderate ; easy 

comment 

16. lecture tape ...following discussion shallow ; moderate ; deep 

comment 

17. lecture tape ...relevant to stated objectives ; irrelevant 

comment 

18. lecture tape ...relevant to participants* needs ; irrelevant 

comment 



GENERAL COMMENTS: 
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OBSERVATION SCHEDULE - B 



Date Time Session 

Group Work Session Individual Work Session 

generally inconclusive generally 
yes no 



1. Did the par ticipants feel that their 
time could have been better spent in 
another activity? 



2. Did the participants feel that they 
were sufficiently involved with the 
expected task? 



3. Did the participants attempt to 

accomplish their assigned task or to 
work on their evaluation plans? 



4. Did the participants believe they 
actually accomplished something 
during this time spot? 



5. Did the participants feel they 

needed more structure for this time? 



6. Did the participants feel they needed 
more guidance or help from the staff 
for this time spot? 
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Formative evaluation is aimed more at long-range generalizations about 
instruction than is summative evaluation. 

One critical task for the evaluator is to combine the judgments of 
merit and shortcoming into a single consensus of program value. 

The educational program having goals that are clearly understood and 
stable is a better program than one having goals that are only 
implicit and changing. 

Educational evaluation is essentially the same as educational research 
in terms of techniques used and in terms of questions to be answered. 

The value of a model such as Bloom's Taxonomy or Stake 1 s ’’countenance 
model” comes in using the categories to sort the different items or 
data after they have been collected. 

It is wrong for the evaluator to try to get the educator to state his 
objectives in terms of student behaviors. 

Item discriminability coefficients should exceed .50 if a 30-item 
test is to have the usually acceptable amount of reliability. 

Questionnaire information is the least reliable and useful informa- 
tion evaluators collect. 

Interviewing as a method of inquiry is universal in the social 
sciences . 

The literature of anthropology serves as an example of the products 
obtained through interviewing informants. 

The following may be obtained from empirical studies and used to 
appraise survey results: 

Estimates of variation between elements in the population and between 
various groupings of these elements. 

Cost factors and analyses, cost relationships,, 

Data of established accuracy for use in testing and correcting 
ordinary procedures. 

The size of samples, method of drawing it, and other features of the 
survey design will not be affected by the kind of analysis to be made 
of the results. 
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16 . 



17 . 



The best starting point for any design is to be found in the aim that 
the survey is to fulfill. 

The simplest and most satisfactory test of the accuracy of an estimate 
from a sample survey is not a direct comparison of the estimate with 
the true value of the variable being estimated. 

A study of attrition rates will be of little help in identifying 
sources of bias. 

Sampling variability is the amount of variability that arises through 
repeated application of a given sampling procedure. 

We cannot ordinarily expect to get very substantial gains in accuracy 
in the estimation of a population proportion through the use of 
stratification. 

Unobtrusive measures compete with formal experimental design to 
provide information to educational decision makers. That is, one 
must choose which has the higher likelihood of reducing error in 
collecting data. 

Quality of teaching as a source of error can be controlled by 
Flander f s interaction analysis for the four groups of sixth graders. 

Archives might include examining science-teacher -of-the-year 
candidates careers. 
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21. Sampling conversation in the teachers 1 lounge is an example of 
simple observation. 



Choice 

22. Which of the following is the outstanding obstacle to representing a 
program* s objectives and priorities? 



a. 

b. 

c . 
d„ 



teachers are not oriented to student behaviors 
goal statements and indicators are oversimplifications 
no educationally meaningful unit of " investment” exists 
goals cannot be represented by numbers, spatial areas, vectors 
pie-graph sectors, etc. 



^23. Interviews typically yield subjective data--descr iptions of the world 
of experience --for which of the following? 

a. goals 

i b. perceptions 

„ c. attitudes 

d. all of the above 
, e. none of the above 

' ,/W3) 
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24. The Chi square technique is commonly used for 

a. describing groups in terms of "fine measurement" data 

b. testing hypotheses regarding "fine measurement" data 

c. describing groups in terms of frequency counts 

d. testing hypotheses regarding frequency counts 

25. The Q Technique and conventional factor analysis are both techniques 
for 

a. analyzing profiles of students 

b. clustering "like things" together 

c. comparing large numbers of groups 

d. evaluating instructional television 

26. The Q sort and the method of paired comparison are both methods which 
could be used for 

a. assigning "priority values" to educational goals 

b. measuring problem solving in students 

c. designing a feedback loop for instruction 

d. testing hypotheses 

27. The process of generalizing from sample data to population conditions 
while at the same time specifying the investigator's confidence in 
drawing correct conclusions is known as 

a. summative evaluation 

b. * interaction analysis 

c. statistical inference 

d. taking a calculated risk 

28. Which of the following is usually not considered a major area of 
specialization for the educational research methodologist? 

a. measurement, testing, instrumentation 

b. research design, experimental controls 

c. statistical description and inference 

d. cost-benefit analysis, program evaluation 

29. "In a statistics -book table of Chi square values, the entries in the 
.05 column indicate the boundary point between the 95% most likely 
Chi square values to be obtained from sample data and the 5% least 
likely Chi square values to be obtained from sample data" 

The previous statement is true only if the samples are randomly drawn 
from a population where 

a. the "null hypothesis" is true 

b. the "null hypothesis" is false 

c. all variables are interrelated 

d. no subgroups (samples) have any meaning 
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It is usually not practical to use the method of paired comparisons 
unless the number of stimulus objects (things to be scaled) is 

a. one 

b. two 

c« four to twelve 

d. twenty to one hundred 

e. at least two hundred 



31. When using a rating scale, the observer 



a. measures behavior by questioning 

b. measures behavior by recording behavioral events 

c. measures behavior by noting degrees of behavior 

d. measures behavior by short time samples 



itch each entry on the right with one of the three entries on the left by putting 
letter in the blank. 



5int of View on Evaluation 



Emphasis 



Self study, motivate self -correction 

Visitation by group of peers 

Control groups, control variables 

Correlation among student talents 

The differences among individual 
students 

The traditional subject-matter 
disciplines 

Prediction of later student success 
Comparison of educational "treatments” 
Norm groups, percentile scores 

Writings 

Campbell and Stanley in the Gage 
Handbook 

Thurstone on Test Theory 

’'National Study’s” Evaluative 
Criteria 

Tyler on the Eight Year 
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Evaluation Institute 
Urbana, Illinois 
July 29-August 9, 1968 



ATTITUDE INVENTORY 



l 
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Directions: For each of the statements below, mark the letter which indicates 

your agreement or disagreement with the statement according to the following code 



1. 



2 . 



3. 



] 

1 



4. 



8. 



9. 



SA 

A 

• 

D 

SD 



I strongly agree with the statement 
I am in slight agreement with the statement 
I am undecided 

I am in slight disagreement with the statement 
I strongly disagree with the statement 



The role of the evaluator should be that of a 
describer rather than a grader . 



The evaluator should determine whether the goals 
of a program are worthwhile. 



Most decisions made in the public schools today 

are based on hunches, hearsay, and individual beliefs. 



Findings from laboratory studies seldom are applicable 
to regular classroom activities. 



One of the first things an evaluator must do is 
obtain a list of behavioral objectives. 



A major role of the evaluator is to make explicit 
the standards by which an educational 
judged. 



program is 



10 . 



Evaluators often pay too much attention to what they 
have been urged to look at, and too little attention 
to other facets. 



The kind of date gathered in an evaluation should 
seldom be determined by what the groups are like 
that will receive the results of the evaluation. 



As long as hoped for outcomes occur, it is not 
important that objectives be stated clearly. 



The most important use of evaluation findings 
is to change the program. 




31 

SA 

46 

19 

SA 

8 

42 

SA 

46 

8 

SA 

4 

12 

SA 

19 



31 

SA 

27 



35 

SA 

23 



12 

SA 

4 

0 

SA 

8 

4 

SA 

0 



50 8 

A ? 
42 0 



31 

A 

31 

31 

A 

42 

23 

A 

15 

42 

A 

50 



42 

A 

42 



42 

A 

58 



8 

A 

12 

4 

A 

15 

15 

A 

31 



12 

15 

12 

4 

4 

8 

23 

8 



4 

8 



12 
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It is possible to evaluate a program without 
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knowing the goals of the individual teachers. 
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The personal characteristics of the evaluator 
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are a major determinant of the evaluation. 
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It is not practical to draw conclusions in 














evaluating a program prior to the program’s 
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We can tell if an educational program is 














successful only by observing whether hoped 
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for changes are occurring in the students. 
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In order to evaluate a program, equal resources 














should be devoted to what teaching is occurr- 
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ing as well as what learning is occurring. 
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It is up to the local educator to rule out 














the study of a variable because it is not 
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one of his objectives. 
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No school can evaluate the impact of its 














program without knowledge of what other 
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The most appropriate instruments for 














evaluating educational programs are 
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standardized tests. 
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Joyous distrust is a sign of health. 
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Everything absolute belongs to pathology. 
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An evaluator has the right to decide 
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what to evaluate. 
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The task of describing curricular objectives 
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is the responsibility of the evaluator. 
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The evaluator should Identify unanticipated 
outcomes of the program. 



It is more important to compare local data 
with national norms than to compare it with 
local norms . 



Absolute standards, e.g. the judgments of 
people, should not be applied to a program. 



In selecting variables for evaluation, the 
evaluator must make a subjective decision. 



The most important use of evaluation findings 
is to justify the program to other groups. 
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PARTICIPANT OPINIONAIRE 



Evaluation Workshop 
University of Illinois 
Urbana, Illinois 



Now that this Workshop is drawing to a close, we are certain that you have 
some reactions as to what parts have been most valuable to you and what 
parts might have been different. This form is designed to make it easy 
for you to pass these reactions along to the workshop planners. It is 
important that every participant complete and return the opinionaire so 
that the reactions of the total group will be reflected# 



The questions are designed to make it easier for you to express your 
reactions. If they do not provide sufficient opportunity, please write 
your comments in your own words. You do not need to indicate your name. 

1. Did you have enough information about this workshop before 
you arrived? Yes , ( } 






2. (If no) What else would you like to have known about? 



3. There are many parts of a Workshop experience that can either 
contribute to your satisfaction or detract from it. For each 
of the following, would you let us know how satisfied you ve 

been? 



a. meals 

Really outstanding 
Very satisfactory 
Just acceptable 
Need improvement 

b. hotel rooms 

Really outstanding 
Very satisfactory 
Just acceptable i 

Need improvement 

c. meeting rooms 

Really outstanding 
Very satisfactory 
Just acceptable 
Need improvement 

d. other facilities or services 

Really outstanding 
Very satisfactory 
Just acceptable 
Need improvement 



1 ( 
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1 ( 
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3 ( 

4 ( 

1 ( 
2 ( 

3 ( 
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) 
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e. facilities for working 

Really outstanding 
Very satisfactory 
Just acceptable 
Need improvement 

f. opportunity for discussion 

Really outstanding 
Very satisfactory 
Just acceptable 
Need improvement 

g. presentations in general 

Really outstanding 
Very satisfactory 
Just acceptable 
Need improvement 



1 ( ) 

2 ( ) 

3 ( ) 

4 ( ) 

1 ( ) 

2 ( ) 

3 ( ) 

4 ( ) 

1 ( ) 

2 ( ) 

3 ( ) 

4 ( ) 



(If you have checked "need improvement" for any of the foregoing, please 
note below any suggestions you may have.) 



4, Would you describe the one or two most valuable ideas that you 
received from attending the Workshop? 






T 



5. As far as you’re concerned, what would have most improved the 
Workshop? 



T 



T 



6. Which one of these phrases best states how related this workshop 
was to your interests and background? 



a. It was over my head 

b. I understood almost everything but the 
conference missed my main interests 

c. It dealt with my main interests in an 
understandable and interesting way 

d. It was too basic, few if any new ideas 
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1 ( ) 

2 ( ) 

3 ( ) 

4 ( ) 



7 . 



Which one of the following statements comes closest to stating your* general 
reaction to the total Workshop? 



The most valuable educational experience of my life 1 ( ) 

An outstanding program, I received much from it....*.. 2 ( ) 

Many parts were valuable, others not very 3 ( ) 

I gained something from attending but less than I expected. ... 4 ( ) 

It was almost a complete waste of time 5 ( ) 

(other) 6 ( ) 



8. After this Workshop is over, is there anything related to the Workshop 
topics that you would like to know more about or to study further? 

Yes 1 ( ) 

No 2 ( ) 

9. (If yes) What specifically would you like to study? 



10. (If yes) How would you like to do so? 

Study on my own i ' 

Attend a class that meets weekly 2 ( ) 

Attend another Workshop * 3 ( ) 

Take a course by correspondence 4 ( ) 

In a local study group • • • 5 ( ) 

t 

(Other) .... » 6 ( ) 



If you have further comments on the Workshop, please write them in your own 
words . 



Evaluation Institute 
Urbana, Illinois 
July 29-August 9, 1968 



Participant Critique Form 

Directions: Please respond with a word, a phrase, or one or more sentences to as 

many of the following questions as you can. Your frank and honest evaluation can 
only benefit everyone concerned. Do not identify yourself by name unless you 

prefer to do so. 



Environment and Facilities 

1. a. To what extent did the relative unavailability of books and journals inter 
fere with your attempts to master the content of this session? 



b. To what extent did reproduced materials given to you by the staff improve 
matters? 



2. a. Did you feel that you lacked a "place to work,” either alone or in small 
groups? 



b. If you had a room at the Union, was it satisfactory? 



c. If you did not have a room at the Union, did your staying elsewhere make 
the Institute any more or less worthwhile to you? 



3. a. Which features of the meeting rooms were inadequate or not conducive to 
learning? 



b. Which features were especially facilitative in the same regard? 
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Scheduling and Organization 



A. a. Was two weeks too long a period to leave your work at home for the purpose 
of attending this session? 



b. Was two weeks too short a period in which to learn much of the content of 
this session? 



5. a. Were you allowed enough time in which to pursue activities of your own 
choosing? 



b. Would you have preferred not to meet in the evening after dinner? 



c. Would fewer meetings per day have been preferable? 



d. Would you have preferred more meetings per day than there actually were? 



6. a. Were the individual lectures too long to sit and listen or take notes? 



b. Were the lectures scheduled in an appropriate sequence? 

i 



7. a. Did you have sufficient opportunities to interact with other participants? 



8. a. Were the instructors too inaccessible or unapproachable so that you 
did not get the individual attention that you desired? 
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* * 



b. Would It have been advisable to have had a few highly-trained graduate 
student assistants present from whom you could have obtained help on 
Individual problems? 



c. Were the staff members helpful In any way? 



9, a. Did the attempts to evaluate your progress and reactions during the 
session (and at this moment) interfere with your work here? 



b. Do you begrudge the time you have spent here answering such questions as 
these on this critique? 



10* In general , was the Institute well organized? 



Content and Presentation 



11. a. Did the content of the lectures and readings presuppose far more 
previous training (in math and statistics) than you had? 



b. Should less training in these areas or more have been presupposed? 



12. To what extent was the content of the lectures and readings relevant to 

what you hoped to accomplish during the session? 



13. Do not be reluctant to single out a staff member for praise or censure. 
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a. Were the lecturers stimulating and interesting? 



b. Were the lecturers competent to speak on the subject assigned them? 



c, Were the lecturers well prepared? 



14. 



Were you disappointed in any way with the group of participants? 



Answer each of the following only by checking the more appropriate blank: 

15. If you had it to do over again would you apply for this Institute which you 

have just completed? Yes _____ ____ No . 

16. If ai Institute such as this is held again would you recommend to others like 
you that they attend? Yes ________ No __ ___ __ 

17. Do you anticipate maintaining some sort of contact with at least one member of 

of the Institute staff? Yes _ No _____ ___ 

18. Do you feel that your understanding of evaluation has been considerably enriched 
in these two weeks? Yes ___ ___ No 

t 

19. Is it likely that you will consult in evaluation with someone else attending 

this institute? Yes No ’ 

20. Would you say that because of this Institute you are more able to state a 
given evaluation problem in operational form so that it is, if it can be, 
amenable to solution? Yes _____ No _______ 

21. Do you feel that the staff should feel that it has accomplished its objectives 
during this two week Institute? Yes _____ No _____ 

Use the remaining space, if you wish, to give us your ideas on what was wrong with 

this session, or what was particularly commendable in it, or how it could have 

been done better. Try particularly to mention items which were not dealt with 

in the questions on the preceding pages. 



ERIC 
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Evaluation Institute 
Urbana, Illinois 
July 29 - August 9, 1968 



Participant Interview Schedule* 
Part I 
(1st half) 



Date Administered 
Name of Interviewer 



Introduction 



1. Identify yourself if it is necessary. 



2. Purpose: 



3. Anonymity : 

4. Begin: 



The reason that I have asked to talk with you has to do with 
your general reaction to the institute so far. The other 
interviewers and I are gathering this type of information 
so that the staff can better organize next week's activities 
as well as evaluate the overall training experience. While 
some things cannot be changed in this institute, I*m sure 
that all of your comments will be useful for designing future 

training programs of this type. 

< 

Your name will not be placed on this interview form. 

Do you have any questions before we begin? 



Institute Design 

1. What has been the most beneficial to you in the institute so far? 



Could you indicate why this is so? 



*EXPLORE EACH ITEM AS FULLY A 3 POSSIBLE BY ASKING SUCH QUESTIONS AS, "IS THERE ANY- 
THING ELSE?", "ANY OTHER IDEAS YOU WANT TO MENTION?", ETC. ' 



o 
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2. Is there anything you would like to see happen more often? Yes No 

IF YES AND NO ELABORATION - What would that be? 



3» In terms of the amount of time spent for activities such as lectures, 
structured groups, work sessions, video viewing, would you like to 
see the proportion of time alloted for these activities changed in 
any way? Yes No 

IF YES - In what way? 



Lectures 

4. What is your general impression of the lectures so far? 

COMMENTS 



POSITIVE 



NEGATIVE 



Do the lectures seem relevant to the other institute activities in which 
you are involved? Yes No 

IF YES - In what ways do the lectures seem relevant? 



IF NO - What could make the lectures more relevant? 
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6 . 



Are there any aspects of the lectures which make them confusing or 
difficult to understand? Yes No 

IF YES - What aspects? 



What could members of the staff do to Improve this situation? 



IF NO - Are there any other comments you would like to make about the lectures? 



Video Tapes 

7, What Is your general Impression of the video-tapes you have seen? 

COMMENTS 



POSITIVE 



NEGATIVE 



What would be your major criticism of the video-tapes? 



CONSIDER CATEGORIES BELOW FOR CLASSIFYING STATEMENTS 

AWARENESS 
PHYSICAL QUALITY 
CONTENT QUALITY 
UNDERSTANDABILITY 
PRACTICALITY 
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Materials 



9. Are the materials, such as the books, papers, evaluation plans, and 
(statistical exercises) of any help to you? Yes No — __ _ 

IF YES - Which of these materials seem to be the most helpful to you: 



How were they helpful? 



10. What materials seem to be of little or no help to you? 



o 

ERIC 



IF MATERIALS ARE INDICATED - Why does this seem to be the case; 



11. What kinds of materials should have been provided which were not made available? 



RECORD "WHY" IF SPECIFIED 



Transferability 
12. You mentioned that 



, and 



were helpful 



xou uusntiuucu tiiou . » » ■ ; t: 

to you (or you liked them). Of these and other activities that you men- 

tioned, do you believe they are presented in such a way that they will 
be helpful to you in your own situation back home? Yes _ No 

IF YES - Which ones will be helpful? — — 



Why? 
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13. Are there any (other) things occurring in this institute that you will 
find useful back home? Yes _____ No _____ 

IF YES - What? 

14. Are there some parts of the institute that you won’t be able to use in 

your own situation back home? Yes No 

IF YES - Which parts - IF NOT ELABORATED 



RECORD "WHY" IF SPECIFIED 



Summary 



15. Is there anything else the institute staff should know, so they might improve 
this experience for you? Yes _____ No 

IF YES - What would that be? 



If you were going to conduct an evaluation institute similar to this one, 
what changes might you make (other than what you have already indicated)? 



GENERALLY REVIEW ALL OF THE RESPONSES CHECKING FOR CORRECTNESS OF 
INFORMATION AND ANY FORGOTTEN IMPRESSIONS. 



"As I mentioned at the beginning of our talk, this information will be 
very helpful to the staff in making decisions about next week's activities 
as well as the designing of future training programs. Thank you for 
your time." 



(A-90) 



o 



Evaluation Institute 
Urbana, Illinois 
July 29-August 9, 1968 



Participant Interview Schedule* 
Part 11 
(2nd half) 



Date Administered 
Name of interviewer 



Introduction 



1* Identify yourself if it is necessary. 

2. Purpose : The reason that 1 have asked to talk with you has to do with 

your general reaction to the institute so far. The other 
interviewers and 1 are gathering this type of information so 
that the staff can better evaluate the overall training ex- 
perience. While some things cannot be changed in this in- 
stitute, I'm sure that all of your comments will be useful 
for designing future training programs of this type. 



3. Anonymity : Your name will not be placed on this interview form. 

4. Begin : Do you have any questions before we. begin? 



Institute Design 

1. What has been the most beneficial to you in the institute? 



Could you indicate why this is so? 



* EXPLORE EACH ITEM AS FULLY AS POSSIBLE BY ASKING SUCH QUESTIONS AS, "IS THERE 
ANYTHING ELSE?", "ANY OTHER IDEAS YOU WANT TO MENTION?", ETC . 
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2. Is there anything you would like to have seen happen more often? Yes _____ No 
IF YES AND NO ELABORATION - What would that be? 



Is there any particular reason why you would like to have seen this happen 
more often? Yes No 



3. In terms of the amount of time spent for activities such as lectures, 
structured groups, work sessions, video viewing, would you h&ve liked 
to see the proportion of time alotted to the activities changed in any 
way ? Yes _____ No _____ 

IF YES - In what way ' 



Lectures 

4. What was your general impression of the lectures? 

COMMENTS 



Positive 



Negative 



5. Dxd the lectures seem relevant to the other institute activities in which 
you were involved? Yes _____ No 

IF YES - Did the lectures seem relevant? 



IF NO - What would have m&de the lectures more relevant? 



6. Were there any aspects of the lectures which made them confusing or difficult 
to understand? Yes No 

IF YES - What aspects __ ______ 

What could members of the staff have done to Improve this situation? 



IF NO - Are there any other comments you would like to make about the lectures? 



Video Tapes 

7. What was your general Impression of the video-tapes you have seen? 

COMMENTS 

Positive Negative 



8. What would be your major criticism of the video tapes? 



CONSIDER BELOW CATEGORIES FOR CLASSIFYING STATEMENTS, 

AWARENESS 

PHYSICAL QUALITY 
CONTENT QUALITY 
UNDERSTANDABILITY 



PRACTICALITY 



Materials 



9* Were the materials such as books , papers, evaluation plans, and (statistical 
exercises) of any help to you? Yes _____ No _____ 

IF YES Which of these materials seem to have been the most helpful to you? 



How were they helpful? 



10, What materials seemed to be little or no help to you? 



IF MATERIALS INDICATED - Why does this seem to be the case? 



11* What kinds of materials should have been provided which were not made 
available? 



RECORD "WHY" IF SPECIFIED 



Transferability 

12. You mentioned that , * an< j 

were helpful to you, or you liked them. Of these and others that you 
mentioned, do you believe they were presented in such a way that they will 
be helpful to you in your own situation back home? Yes _____ No 

IF YES - Which ones will be helpful - 

In what way? 
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13 . 



Were there any (other) activities occurring in this institute that you 
will find useful back home? Yes No 

IF YES - What 



14. Are there some parts of the institute that you won't be able to use in 
your own situation back home? Yes _____ No . 

IF YES - Which parts - IF NOT ELABORATED - 



RECORD "WHY" IF SPECIFIED m 



Summary 

15. Is there anything else the institute staff should have known, so they might 
have improved this experience for you? Yes No 

IF YES - What 



16. If you were going to conduct an evaluation institute similar to this one, 
what changes might you make (other than what you have already indicated)? 



(GENERALLY REVIEW ALL OF THE RESPONSES CHECKING FOR CORRECTNESS OF INFORMATION 
AND ANY FORGOTTEN IMPRESSIONS) 

"As I mentioned at the beginning of our talk, this information will be very 
helpful to the staff in designing future training programs. Thank you for 

your time." 
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